00000080:00200000:2.0:1789431426.531933:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.531937:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.531957:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.531961:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.531971:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.531975:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.531983:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.531987:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.531996:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.531998:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.532007:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.532010:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.532016:0:320464:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.532019:0:320464:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431426.591988:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.592003:0:320471:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.592009:0:320471:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.592029:0:320471:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.592045:0:320471:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.592058:0:320471:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.592066:0:320471:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.592072:0:320471:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.592075:0:320471:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.592082:0:320471:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c213bcb8 00010000:00010000:2.0:1789431426.592087:0:320471:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.592100:0:320471:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.596381:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596392:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.596404:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596407:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596416:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596419:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596427:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596430:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596438:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596440:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596448:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596451:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596462:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596465:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596473:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596475:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596482:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596486:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431426.596675:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596680:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.596689:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596692:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596700:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596703:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596713:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596717:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596725:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596728:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596736:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596738:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596745:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596748:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596755:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596758:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.596765:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596769:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431426.596934:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596940:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.596965:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596969:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596976:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596979:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596985:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596988:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.596995:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.596998:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597005:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597009:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597020:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597022:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597029:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597032:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597040:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597043:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431426.597174:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597180:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.597191:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597194:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597214:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597218:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597230:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597234:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597242:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597244:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597252:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597255:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597264:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597267:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597274:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597277:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597285:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597288:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431426.597409:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597414:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.597422:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597425:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597432:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597435:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597443:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597446:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597453:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597456:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597465:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597467:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597474:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597476:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597483:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597486:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597493:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597497:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431426.597614:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597619:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.597627:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597630:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597639:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597642:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597650:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597652:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597659:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597662:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597670:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597673:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597680:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597683:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597689:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597692:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597699:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597702:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431426.597819:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597824:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.597861:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597865:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597874:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597877:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597884:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597887:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597894:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597896:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.597905:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597907:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597914:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597917:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597924:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597927:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.597934:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.597937:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.604479:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604488:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.604500:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604504:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.604512:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604516:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.604524:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604527:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.604535:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604538:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.604549:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604552:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.604560:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604562:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.604571:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604574:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.604582:0:320471:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.604586:0:320471:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431426.646105:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.646192:0:320476:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.646199:0:320476:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.646290:0:320476:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.646308:0:320476:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.646322:0:320476:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.646329:0:320476:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.646335:0:320476:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.646338:0:320476:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.646343:0:320476:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2113cb8 00010000:00010000:2.0:1789431426.646347:0:320476:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.646354:0:320476:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.658577:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658594:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.658605:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658609:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.658616:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658618:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.658628:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658631:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.658640:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658643:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.658650:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658653:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.658661:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658665:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.658672:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658675:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.658683:0:320476:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.658687:0:320476:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.717570:0:320484:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.717587:0:320484:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.717595:0:320484:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.717622:0:320484:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.717640:0:320484:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.717655:0:320484:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.717663:0:320484:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.717669:0:320484:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.717673:0:320484:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.717679:0:320484:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c215bcb8 00010000:00010000:2.0:1789431426.717683:0:320484:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.717693:0:320484:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.755297:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.755307:0:320492:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.755312:0:320492:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.755332:0:320492:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.755348:0:320492:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.755362:0:320492:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.755370:0:320492:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.755376:0:320492:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.755378:0:320492:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.755384:0:320492:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2163cb8 00010000:00010000:2.0:1789431426.755387:0:320492:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.755395:0:320492:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.756580:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756594:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.756606:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756609:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756617:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756620:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756626:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756629:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756636:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756639:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756645:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756648:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756655:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756658:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756664:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756667:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756673:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756677:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431426.756821:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756826:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.756858:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756862:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756871:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756874:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756884:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756886:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756894:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756897:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.756903:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756906:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756913:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756916:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756922:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756925:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.756931:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.756934:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431426.757112:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757117:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.757124:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757127:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757134:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757137:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757144:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757147:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757153:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757156:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757164:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757167:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757175:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757178:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757185:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757187:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757195:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757198:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431426.757326:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757331:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.757340:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757343:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757350:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757352:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757360:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757363:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757370:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757373:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757379:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757382:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757389:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757392:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757399:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757401:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757407:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757411:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431426.757525:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757530:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.757538:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757540:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757548:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757551:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757557:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757560:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757566:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757568:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757575:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757577:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757583:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757586:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757592:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757594:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757601:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757604:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431426.757710:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757715:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.757724:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757726:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757733:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757735:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757743:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757746:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757753:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757755:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.757761:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757764:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757770:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757773:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757779:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757782:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.757789:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.757792:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431426.757935:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758460:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.758487:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758492:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758502:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758505:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758514:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758517:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758525:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758528:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758534:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758537:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758543:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758546:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758555:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758557:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758564:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758568:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.758742:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758747:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.758756:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758758:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758765:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758768:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758775:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758778:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758786:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758789:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.758797:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758800:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758808:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758811:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758818:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.758821:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.758827:0:320492:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.759916:0:320492:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431426.783550:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783564:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.783575:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783578:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.783585:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783588:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.783596:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783599:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.783608:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783610:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.783616:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783619:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.783626:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783629:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.783638:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783641:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.783649:0:320495:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.783652:0:320495:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.797412:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.797423:0:320498:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.797429:0:320498:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.797453:0:320498:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.797471:0:320498:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.797486:0:320498:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.797494:0:320498:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.797501:0:320498:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.797504:0:320498:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.797510:0:320498:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2123cb8 00010000:00010000:2.0:1789431426.797513:0:320498:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.797522:0:320498:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.798790:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798803:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.798815:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798818:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.798826:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798829:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.798865:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798868:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.798876:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798879:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.798887:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798890:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.798898:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798902:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.798910:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798913:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.798923:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.798926:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431426.799125:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799131:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.799142:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799144:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799154:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799157:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799165:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799168:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799176:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799178:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799187:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799190:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799199:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799202:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799209:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799212:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799221:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799225:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431426.799371:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799377:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.799385:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799388:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799395:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799398:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799405:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799408:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799418:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799421:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799430:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799432:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799439:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799442:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799450:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799453:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799461:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799465:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431426.799581:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799586:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.799594:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799597:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799604:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799606:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799613:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799616:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799623:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799625:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799632:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799635:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799642:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799645:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799653:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799656:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799664:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799668:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431426.799777:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799782:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.799790:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799794:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799801:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799804:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799812:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799815:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799822:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799824:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.799858:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799862:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799873:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799875:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799882:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799884:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.799892:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.799895:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431426.800039:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800044:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.800052:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800055:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.800062:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800064:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.800071:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800073:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.800080:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800082:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.800091:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800094:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.800103:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800106:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.800114:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800117:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.800126:0:320498:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.800130:0:320498:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.832589:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.832606:0:320505:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.832613:0:320505:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.832641:0:320505:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.832659:0:320505:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.832673:0:320505:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.832681:0:320505:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.832688:0:320505:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.832691:0:320505:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.832698:0:320505:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c217bcb8 00010000:00010000:2.0:1789431426.832702:0:320505:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.832712:0:320505:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.834299:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834312:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.834325:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834329:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834337:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834340:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834348:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834352:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834359:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834362:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834369:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834372:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834379:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834382:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834390:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834393:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834400:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834404:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431426.834580:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834585:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.834592:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834595:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834601:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834603:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834608:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834611:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834619:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834621:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834628:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834631:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834637:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834640:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834646:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834648:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834654:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834657:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431426.834798:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834803:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.834814:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834817:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834824:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834827:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834862:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834867:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834875:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834878:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.834884:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834887:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834894:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834896:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834902:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834905:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.834911:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.834915:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431426.835042:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835047:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.835054:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835056:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835063:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835065:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835073:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835076:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835082:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835084:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835090:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835092:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835098:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835100:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835107:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835111:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835117:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835121:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431426.835227:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835232:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.835242:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835245:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835254:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835257:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835265:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835268:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835276:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835279:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.835285:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835288:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835294:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835296:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835303:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835306:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.835312:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.835315:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431426.836427:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836441:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.836453:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836457:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.836464:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836467:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.836475:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836478:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.836487:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836491:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.836500:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836503:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.836511:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836514:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.836523:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836525:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.836532:0:320505:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.836534:0:320505:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.859693:0:320509:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.859710:0:320509:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.859718:0:320509:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.859745:0:320509:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.859763:0:320509:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.859779:0:320509:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.859787:0:320509:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.859794:0:320509:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.859798:0:320509:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.859805:0:320509:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c207bcb8 00010000:00010000:2.0:1789431426.859809:0:320509:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.859817:0:320509:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.874004:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.874022:0:320511:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.874030:0:320511:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.874054:0:320511:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.874072:0:320511:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.874086:0:320511:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.874094:0:320511:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.874100:0:320511:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.874103:0:320511:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.874109:0:320511:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2183cb8 00010000:00010000:2.0:1789431426.874112:0:320511:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.874120:0:320511:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.875508:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875522:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.875531:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875535:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.875542:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875544:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.875550:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875553:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.875559:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875562:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.875570:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875573:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.875581:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875583:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.875589:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875592:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.875598:0:320511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.875601:0:320511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.947644:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431426.947658:0:320525:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431426.947665:0:320525:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431426.947693:0:320525:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431426.947713:0:320525:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.947728:0:320525:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.947736:0:320525:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431426.947754:0:320525:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431426.947758:0:320525:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431426.947764:0:320525:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2133cb8 00010000:00010000:2.0:1789431426.947768:0:320525:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431426.947777:0:320525:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431426.949738:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.951957:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.952026:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952033:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.952044:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952047:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.952055:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952058:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.952066:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952070:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.952079:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952082:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.952091:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952094:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.952100:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952102:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.952109:0:320525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.952112:0:320525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.986053:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986065:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.986076:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986079:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986086:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986090:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986097:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986100:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986108:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986111:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986119:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986122:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986130:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986133:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986142:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986145:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986153:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986157:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431426.986314:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986320:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.986328:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986332:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986340:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986343:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986349:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986352:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986360:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986363:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986372:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986375:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986383:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986386:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986394:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986397:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986408:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986412:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431426.986539:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986544:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.986552:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986555:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986562:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986565:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986572:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986575:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986583:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986587:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986595:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986598:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986606:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986609:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986616:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986619:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986628:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986631:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431426.986748:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986753:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.986762:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986765:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986773:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986776:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986784:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986787:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986796:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986799:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.986808:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986811:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986820:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986824:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986854:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986858:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.986868:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.986871:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431426.987000:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987004:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.987011:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987014:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987021:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987024:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987030:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987033:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987041:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987044:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987051:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987054:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987062:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987065:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987071:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987073:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987080:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987085:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431426.987185:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987190:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.987199:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987202:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987211:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987214:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987223:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987226:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987233:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987235:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987241:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987244:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987250:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987253:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987261:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987264:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987271:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987274:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431426.987376:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987380:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.987388:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987390:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987399:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987402:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987409:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987412:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987419:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987422:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987429:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987431:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987438:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987440:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987448:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987450:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987459:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987462:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431426.987607:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987611:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431426.987620:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987622:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987628:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987632:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987637:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987640:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987646:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987649:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431426.987655:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987658:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987664:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987666:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987672:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987675:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431426.987681:0:320534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431426.987685:0:320534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431427.045391:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.045406:0:320540:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.045413:0:320540:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.045438:0:320540:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.045454:0:320540:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.045469:0:320540:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.045477:0:320540:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.045484:0:320540:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.045487:0:320540:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.045494:0:320540:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c217bcb8 00010000:00010000:2.0:1789431427.045498:0:320540:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.045507:0:320540:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.053228:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053243:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.053255:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053260:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053269:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053271:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053279:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053282:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053289:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053292:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053303:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053306:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053313:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053316:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053327:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053330:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053338:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053341:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431427.053562:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053567:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.053578:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053581:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053588:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053591:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053601:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053604:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053612:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053614:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.053621:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053623:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053630:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053632:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053640:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053642:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.053650:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.053653:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431427.057557:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057574:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.057587:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057591:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.057599:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057602:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.057610:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057613:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.057621:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057625:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.057633:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057636:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.057645:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057648:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.057657:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057659:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.057925:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.057932:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431427.058097:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058103:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.058348:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058354:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058370:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058373:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058380:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058383:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058390:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058393:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058400:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058404:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058411:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058415:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058422:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058425:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058434:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058438:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431427.058560:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058566:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.058575:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058578:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058586:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058589:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058597:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058600:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058607:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058610:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.058618:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058620:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058627:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058629:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058636:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058638:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.058646:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.058649:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431427.059011:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059017:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.059026:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059029:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.059036:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059039:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.059046:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059049:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.059056:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059058:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.059066:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059069:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.059078:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059081:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.059089:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059092:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.059101:0:320540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.059104:0:320540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.078668:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.078687:0:320544:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.078695:0:320544:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.078721:0:320544:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.078738:0:320544:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.078754:0:320544:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.078762:0:320544:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.078769:0:320544:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.078772:0:320544:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.078779:0:320544:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c21dbcb8 00010000:00010000:2.0:1789431427.078783:0:320544:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.078808:0:320544:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.080163:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080175:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.080187:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080191:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.080199:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080202:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.080209:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080212:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.080221:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080224:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.080233:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080237:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.080245:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080248:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.080256:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080259:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.080267:0:320544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.080271:0:320544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.098291:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.098302:0:320546:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.098309:0:320546:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.098333:0:320546:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.098351:0:320546:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.098372:0:320546:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.098382:0:320546:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.098388:0:320546:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.098392:0:320546:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.098398:0:320546:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c218bcb8 00010000:00010000:2.0:1789431427.098402:0:320546:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.098411:0:320546:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.101086:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101093:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.101102:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101105:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.101113:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101116:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.101122:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101125:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.101134:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101137:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.101145:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101147:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.101156:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101159:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.101169:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101172:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.101181:0:320546:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.101185:0:320546:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.141631:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.141649:0:320551:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.141656:0:320551:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.141683:0:320551:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.141702:0:320551:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.141718:0:320551:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.141726:0:320551:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.141733:0:320551:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.141736:0:320551:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.141755:0:320551:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c21f3cb8 00010000:00010000:2.0:1789431427.141760:0:320551:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.141769:0:320551:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.144650:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144666:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.144679:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144683:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.144693:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144697:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.144706:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144709:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.144717:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144720:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.144727:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144731:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.144739:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144757:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.144769:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144772:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.144781:0:320551:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.144785:0:320551:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.165588:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165603:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.165614:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165617:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.165626:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165630:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.165638:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165641:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.165650:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165653:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.165661:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165665:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.165672:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165675:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.165682:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165685:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.165695:0:320557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.165699:0:320557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.185265:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.185274:0:320560:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.185280:0:320560:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.185304:0:320560:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.185321:0:320560:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.185336:0:320560:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.185344:0:320560:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.185351:0:320560:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.185354:0:320560:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.185360:0:320560:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2183cb8 00010000:00010000:2.0:1789431427.185364:0:320560:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.185373:0:320560:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.187815:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.187826:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.188096:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188102:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.188113:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188117:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.188126:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188128:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.188135:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188138:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.188145:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188149:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.188156:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188159:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.188167:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188170:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.188177:0:320560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.188181:0:320560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.208344:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.208362:0:320565:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.208369:0:320565:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.208396:0:320565:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.208414:0:320565:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.208429:0:320565:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.208438:0:320565:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.208445:0:320565:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.208448:0:320565:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.208455:0:320565:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c21cbcb8 00010000:00010000:2.0:1789431427.208459:0:320565:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.208467:0:320565:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.209773:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209787:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.209799:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209803:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.209811:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209814:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.209822:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209825:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.209860:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209865:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.209874:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209878:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.209887:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209889:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.209897:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209901:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.209910:0:320565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.209913:0:320565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.218330:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.218349:0:320568:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.218357:0:320568:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.218383:0:320568:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.218401:0:320568:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.218418:0:320568:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.218433:0:320568:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.218441:0:320568:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.218445:0:320568:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.218452:0:320568:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c21abcb8 00010000:00010000:2.0:1789431427.218456:0:320568:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.218464:0:320568:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.219721:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219733:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.219744:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219747:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.219753:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219756:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.219762:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219764:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.219771:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219773:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.219780:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219783:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.219790:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219793:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.219800:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219803:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.219811:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.219815:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431427.220037:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220043:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.220053:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220056:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220065:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220068:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220075:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220078:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220086:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220089:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220097:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220100:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220108:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220111:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220117:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220120:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220126:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220130:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431427.220274:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220279:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.220288:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220291:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220300:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220303:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220311:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220314:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220321:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220323:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220332:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220335:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220344:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220347:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220355:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220358:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220367:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220370:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431427.220497:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220502:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.220513:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220516:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220525:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220528:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220536:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220538:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220547:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220550:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220557:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220560:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220568:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220571:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220578:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220580:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220588:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220592:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431427.220704:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220709:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.220717:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220720:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220727:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220730:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220737:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220740:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220748:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220750:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.220760:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220763:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220771:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220774:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220782:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220785:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.220794:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.220797:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431427.223121:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223137:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.223148:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223152:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.223160:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223163:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.223170:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223173:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.223179:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223182:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.223188:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223191:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.223197:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223200:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.223207:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223209:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.223218:0:320568:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.223221:0:320568:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.290881:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.290978:0:320574:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.290987:0:320574:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.291016:0:320574:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.291034:0:320574:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.291050:0:320574:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.291058:0:320574:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.291065:0:320574:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.291068:0:320574:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.291075:0:320574:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2213cb8 00010000:00010000:2.0:1789431427.291078:0:320574:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.291086:0:320574:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.299734:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299750:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.299760:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299764:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.299770:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299773:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.299780:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299783:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.299790:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299793:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.299802:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299806:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.299814:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299817:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.299823:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299826:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.299868:0:320574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.299872:0:320574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.312495:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312511:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.312526:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312530:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.312538:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312541:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.312550:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312553:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.312561:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312565:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.312572:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312576:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.312588:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312591:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.312599:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312602:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.312609:0:320576:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.312613:0:320576:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431427.375412:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.375425:0:320582:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.375430:0:320582:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.375444:0:320582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.375457:0:320582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.375473:0:320582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.375480:0:320582:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.375484:0:320582:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.375487:0:320582:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.375492:0:320582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2243cb8 00010000:00010000:2.0:1789431427.375496:0:320582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.375504:0:320582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.380442:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380459:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.380474:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380478:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380486:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380489:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380497:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380500:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380509:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380512:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380520:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380524:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380534:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380537:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380548:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380551:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380562:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380566:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431427.380772:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380777:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.380788:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380791:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380800:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380804:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380812:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380814:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380821:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380824:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.380854:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380859:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380871:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380874:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380883:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.380887:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.380895:0:320582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.383325:0:320582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431427.419670:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420062:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.420081:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420086:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.420095:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420097:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.420105:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420107:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.420115:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420118:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.420127:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420130:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.420139:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420142:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.420148:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420151:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.420159:0:320586:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.420163:0:320586:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.473674:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.473690:0:320595:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.473697:0:320595:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.473719:0:320595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.473735:0:320595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.473750:0:320595:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.473758:0:320595:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.473765:0:320595:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.473769:0:320595:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.473775:0:320595:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2243cb8 00010000:00010000:2.0:1789431427.473780:0:320595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.473789:0:320595:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.476652:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476661:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.476672:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476676:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.476684:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476688:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.476695:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476698:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.476706:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476709:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.476795:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476800:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.476813:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476816:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.476826:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476829:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.476855:0:320595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.476859:0:320595:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.525757:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.525776:0:320600:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.525785:0:320600:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.525825:0:320600:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.525872:0:320600:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.525889:0:320600:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.525904:0:320600:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.525911:0:320600:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.525915:0:320600:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.525922:0:320600:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c226bcb8 00010000:00010000:2.0:1789431427.525926:0:320600:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.525947:0:320600:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.528227:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528242:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.528260:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528264:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.528272:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528274:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.528281:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528284:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.528292:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528295:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.528302:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528305:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.528313:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528316:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.528322:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528325:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.528332:0:320600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.528335:0:320600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.540206:0:320603:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.540223:0:320603:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.540231:0:320603:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.540255:0:320603:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.540272:0:320603:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.540288:0:320603:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.540296:0:320603:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.540303:0:320603:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.540307:0:320603:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.540313:0:320603:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2203cb8 00010000:00010000:2.0:1789431427.540317:0:320603:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.540326:0:320603:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.621578:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621596:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.621610:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621615:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.621625:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621628:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.621638:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621641:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.621649:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621653:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.621663:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621666:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.621675:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621678:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.621686:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621689:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.621697:0:320607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.621700:0:320607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.639022:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.639038:0:320610:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.639045:0:320610:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.639071:0:320610:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.639089:0:320610:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.639105:0:320610:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.639113:0:320610:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.639119:0:320610:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.639122:0:320610:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.639129:0:320610:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c221bcb8 00010000:00010000:2.0:1789431427.639133:0:320610:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.639153:0:320610:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.640437:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640449:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.640459:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640463:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640471:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640474:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640482:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640485:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640492:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640495:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640503:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640506:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640513:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640516:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640524:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640527:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640533:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640537:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431427.640714:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640719:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.640728:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640731:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640739:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640742:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640750:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640754:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640763:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640766:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.640774:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640778:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640786:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640788:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640795:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640798:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.640805:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640809:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431427.640988:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.640993:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.641004:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641007:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641015:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641019:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641026:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641029:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641036:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641039:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641046:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641049:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641056:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641059:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641066:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641069:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641077:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641081:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431427.641204:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641209:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.641218:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641221:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641228:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641230:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641238:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641241:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641247:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641250:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641258:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641260:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641268:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641271:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641277:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641279:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641286:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641290:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431427.641400:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641405:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.641416:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641419:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641426:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641428:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641435:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641438:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641445:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641448:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641455:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641458:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641465:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641468:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641475:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641478:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641487:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641490:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431427.641620:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641626:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.641635:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641638:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641646:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641649:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641656:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641659:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641666:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641668:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.641675:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641678:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641685:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641688:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641696:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641700:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.641707:0:320610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.641711:0:320610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.679412:0:320614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.679430:0:320614:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.679437:0:320614:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.679465:0:320614:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.679485:0:320614:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.679503:0:320614:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.679511:0:320614:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.679518:0:320614:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.679522:0:320614:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.679529:0:320614:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2273cb8 00010000:00010000:2.0:1789431427.679532:0:320614:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.679540:0:320614:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.696636:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.696645:0:320616:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.696651:0:320616:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.696676:0:320616:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.696695:0:320616:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.696710:0:320616:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.696718:0:320616:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.696725:0:320616:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.696728:0:320616:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.696735:0:320616:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2263cb8 00010000:00010000:2.0:1789431427.696739:0:320616:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.696749:0:320616:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.699558:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699570:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.699581:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699585:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.699594:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699596:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.699607:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699611:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.699620:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699623:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.699631:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699634:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.699641:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699643:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.699649:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699651:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.699657:0:320616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.699660:0:320616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.743339:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.743357:0:320627:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.743365:0:320627:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.743394:0:320627:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.743413:0:320627:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.743428:0:320627:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.743435:0:320627:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.743443:0:320627:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.743446:0:320627:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.743454:0:320627:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2203cb8 00010000:00010000:2.0:1789431427.743458:0:320627:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.743467:0:320627:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.744901:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744915:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.744925:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744929:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.744952:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744956:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.744966:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744969:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.744976:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744978:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.744986:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744989:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.744996:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.744999:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.745006:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.745009:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.745016:0:320627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.745020:0:320627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.759692:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.759704:0:320632:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.759710:0:320632:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.759725:0:320632:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.759739:0:320632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.759750:0:320632:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.759758:0:320632:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.759762:0:320632:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.759765:0:320632:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.759770:0:320632:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c229bcb8 00010000:00010000:2.0:1789431427.759773:0:320632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.759782:0:320632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.761635:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761648:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.761657:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761661:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.761668:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761670:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.761677:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761680:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.761686:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761689:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.761696:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761699:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.761706:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761709:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.761716:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761719:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.761726:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761729:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431427.761905:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761910:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.761918:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.761921:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.761933:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762658:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762683:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762687:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762694:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762697:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762706:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762709:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.762718:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762721:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.762730:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762733:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.762740:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762744:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431427.762929:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762945:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.762956:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762959:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762966:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762969:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762976:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762979:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762986:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762989:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.762995:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.762998:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.763553:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.763562:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776130:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776136:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776149:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776153:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431427.776289:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776294:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.776301:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776304:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776311:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776314:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776320:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776323:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776330:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776332:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776339:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776342:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776348:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776351:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776360:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776363:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776370:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776373:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431427.776499:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776504:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.776511:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776515:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776522:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776524:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776532:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776534:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776541:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776544:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776551:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776554:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776560:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776563:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776570:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776573:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776580:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776583:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431427.776685:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776690:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.776702:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776705:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776712:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776715:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776721:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776724:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776732:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776734:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776743:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776746:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776753:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776756:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776763:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776766:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.776775:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776779:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431427.776919:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776925:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.776946:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776949:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776959:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776962:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776970:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776973:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776980:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776983:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.776991:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.776994:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777000:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777003:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777010:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777012:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777019:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777023:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.777205:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777210:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.777221:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777224:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.777231:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777234:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.777242:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777245:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.777253:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777256:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.777263:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777267:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777274:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777276:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777284:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777287:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.777295:0:320632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.777298:0:320632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431427.869984:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870000:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.870013:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870015:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.870025:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870028:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.870035:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870038:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.870047:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870050:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.870058:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870061:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.870067:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870069:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.870075:0:320642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.870078:0:320642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431427.882047:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431427.882066:0:320645:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431427.882073:0:320645:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431427.882097:0:320645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431427.882115:0:320645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.882135:0:320645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.882143:0:320645:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431427.882150:0:320645:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431427.882153:0:320645:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431427.882160:0:320645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22abcb8 00010000:00010000:2.0:1789431427.882164:0:320645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431427.882173:0:320645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431427.887213:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887229:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.887242:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887246:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887255:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887258:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887267:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887270:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887277:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887280:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887288:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887291:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887300:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887302:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887312:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887315:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887323:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887327:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431427.887523:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887529:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.887538:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887541:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887551:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887554:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887562:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887565:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887573:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887576:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.887584:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887587:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887595:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887598:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887604:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887607:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.887614:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.887618:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431427.890722:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890740:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.890753:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890758:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.890768:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890771:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.890779:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890782:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.890788:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890792:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.890800:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890803:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.890812:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890815:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.890826:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890829:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.890972:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.890977:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431427.891120:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891125:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.891134:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891137:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891147:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891151:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891160:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891163:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891170:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891173:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891181:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891184:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891191:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891194:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891202:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891205:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891213:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891217:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431427.891332:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891337:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.891347:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891350:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891359:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891362:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891370:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891373:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891381:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891384:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891392:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891395:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891404:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891407:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891416:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891419:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891427:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891432:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431427.891561:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891567:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431427.891576:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891580:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891588:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891591:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891599:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891602:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891610:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891613:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431427.891623:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891626:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891635:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891637:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891646:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891648:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431427.891656:0:320645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431427.891660:0:320645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.010590:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.010608:0:320659:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.010616:0:320659:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.010643:0:320659:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.010660:0:320659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.010923:0:320659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.010941:0:320659:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.010948:0:320659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.010951:0:320659:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.010958:0:320659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22e3cb8 00010000:00010000:2.0:1789431428.010962:0:320659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.010970:0:320659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.014597:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014613:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.014626:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014630:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.014639:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014642:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.014651:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014654:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.014677:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014681:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.014692:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014696:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.014704:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014707:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.014715:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014718:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.014726:0:320659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.014729:0:320659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.039819:0:320665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.039871:0:320665:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.039880:0:320665:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.039908:0:320665:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.039929:0:320665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.039962:0:320665:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.039971:0:320665:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.039979:0:320665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.039982:0:320665:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.039990:0:320665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22ebcb8 00010000:00010000:2.0:1789431428.039994:0:320665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.040003:0:320665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.044678:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.044697:0:320667:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.044705:0:320667:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.044730:0:320667:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.044748:0:320667:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.044762:0:320667:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.044770:0:320667:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.044776:0:320667:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.044780:0:320667:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.044786:0:320667:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22fbcb8 00010000:00010000:2.0:1789431428.044790:0:320667:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.044798:0:320667:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.051023:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051029:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.051039:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051043:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051052:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051054:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051062:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051065:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051072:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051075:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051083:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051086:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051094:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051097:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051105:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051107:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051116:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051120:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431428.051389:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051394:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.051405:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051409:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051420:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051422:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051431:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051434:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051443:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051446:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051454:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051456:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051464:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051466:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051473:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051475:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051483:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051487:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431428.051623:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051629:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.051637:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051640:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051648:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051650:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051658:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051661:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051668:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051671:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.051678:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051681:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051689:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051692:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051701:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051704:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.051711:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.051714:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431428.054354:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054360:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.054371:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054374:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.054382:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054385:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.054391:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054394:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.054402:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054406:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.054416:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054419:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.054429:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054501:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.054516:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054520:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.054529:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.054532:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431428.054999:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055008:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.055028:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055119:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055134:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055205:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055222:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055226:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055234:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055236:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055245:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055248:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055256:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055259:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055267:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055270:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055278:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055281:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.055397:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055402:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.055410:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055413:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055422:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055424:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055544:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055549:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055560:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055564:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055572:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055575:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055582:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055584:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055592:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055594:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055602:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055606:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431428.055824:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055829:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.055857:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055862:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055870:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055874:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055881:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055883:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055892:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055895:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.055903:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055906:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055913:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055915:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055922:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055925:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.055945:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.055949:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.056296:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056302:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.056312:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056315:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.056323:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056325:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.056332:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056335:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.056342:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056345:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.056352:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056355:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.056361:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056364:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.056373:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056375:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.056383:0:320667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.056386:0:320667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431428.091396:0:320670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.091415:0:320670:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.091423:0:320670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.091461:0:320670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.091478:0:320670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.091493:0:320670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.091502:0:320670:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.091509:0:320670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.091513:0:320670:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.091519:0:320670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c225bcb8 00010000:00010000:2.0:1789431428.091523:0:320670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.091533:0:320670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.106337:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.106358:0:320673:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.106366:0:320673:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.106392:0:320673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.106409:0:320673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.106424:0:320673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.106431:0:320673:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.106439:0:320673:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.106442:0:320673:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.106450:0:320673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22f3cb8 00010000:00010000:2.0:1789431428.106454:0:320673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.106463:0:320673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.109429:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109443:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.109455:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109459:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109467:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109470:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109477:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109480:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109487:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109489:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109496:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109500:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109506:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109509:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109516:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109519:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109526:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109529:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.109702:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109707:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.109715:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109718:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109726:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109728:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109735:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109737:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109744:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109747:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.109756:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109759:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109766:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109769:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109776:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109778:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.109786:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.109789:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.111085:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111098:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.111110:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111114:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111123:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111126:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111132:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111135:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111141:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111144:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111150:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111154:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111161:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111164:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111172:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111175:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111184:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111188:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.111337:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111343:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.111351:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111354:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111361:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111364:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111373:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111376:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111384:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111387:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111394:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111397:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111405:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111407:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111414:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111416:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.111424:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111428:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.111546:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111551:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.111562:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111565:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111572:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.111575:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.111583:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113135:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113163:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113168:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113179:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113182:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113190:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113193:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113201:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113204:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113212:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113216:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431428.113408:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113413:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.113423:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113426:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113448:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113452:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113461:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113464:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113472:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113475:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.113482:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113486:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113495:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113498:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113506:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113509:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.113517:0:320673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.113521:0:320673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.165546:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.165565:0:320680:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.165572:0:320680:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.165597:0:320680:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.165617:0:320680:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.165632:0:320680:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.165641:0:320680:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.165648:0:320680:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.165652:0:320680:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.165659:0:320680:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22ebcb8 00010000:00010000:2.0:1789431428.165663:0:320680:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.165678:0:320680:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.167166:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167179:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.167193:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167198:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167209:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167212:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167225:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167228:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167244:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167247:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167262:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167265:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167281:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167285:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167299:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167303:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167311:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167315:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.167509:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167514:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.167524:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167527:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167534:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167537:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167546:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167550:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167558:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167561:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167569:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167572:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167581:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167583:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167592:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167595:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167602:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167606:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.167755:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167760:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.167770:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167774:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167782:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167785:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167793:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167796:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167803:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167806:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.167813:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167816:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167823:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167826:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167861:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167865:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.167874:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.167878:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.168022:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168028:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.168037:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168040:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168048:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168051:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168058:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168061:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168068:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168071:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168077:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168080:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168086:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168089:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168097:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168100:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168107:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168111:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.168216:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168221:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.168230:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168233:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168240:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168243:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168251:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168254:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168261:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168264:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168271:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168274:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168281:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168284:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168292:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168296:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168303:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168307:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431428.168432:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168438:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.168448:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168451:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168459:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168463:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168471:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168474:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168482:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168486:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.168494:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168497:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168506:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168508:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168516:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168519:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.168528:0:320680:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.168532:0:320680:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.218077:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218090:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.218160:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218165:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218178:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218181:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218192:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218196:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218209:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218212:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218224:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218228:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218236:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218239:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218246:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218315:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218333:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218337:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431428.218541:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218547:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.218558:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218561:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218571:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218574:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218583:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218586:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218595:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218598:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218605:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218608:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218616:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218618:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218624:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218627:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.218634:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218638:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431428.218857:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218863:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.218882:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218885:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218906:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218911:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218929:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218945:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218963:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218967:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.218987:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.218990:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219010:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219014:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219033:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219036:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219056:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219060:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431428.219440:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219446:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.219457:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219460:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.219470:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219473:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.219482:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219484:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.219492:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219495:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.219502:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219505:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219513:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219516:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219525:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219528:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.219534:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219537:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431428.219663:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.219669:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.219682:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.220202:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225167:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225185:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225211:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225215:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225227:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225231:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225260:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225264:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225277:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225280:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225288:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225291:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225300:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225304:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.225525:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225531:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.225541:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225544:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225551:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225554:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225561:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225564:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225571:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225574:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225582:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225585:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225598:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225602:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225610:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225613:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225624:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225628:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431428.225761:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225766:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.225779:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225783:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225796:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225800:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225812:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225815:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225825:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225829:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.225878:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225882:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225894:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225899:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225913:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225916:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.225924:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.225929:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.226122:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226127:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.226136:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226139:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.226146:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226149:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.226156:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226159:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.226166:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226170:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.226176:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226179:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.226186:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226189:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.226196:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226199:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.226206:0:320688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.226210:0:320688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431428.300976:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.300990:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.301002:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301006:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.301013:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301016:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.301023:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301026:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.301034:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301037:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.301045:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301048:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.301055:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301058:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.301065:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301068:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.301075:0:320694:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.301079:0:320694:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.318539:0:320700:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.318558:0:320700:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.318565:0:320700:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.318591:0:320700:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.318610:0:320700:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.318625:0:320700:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.318634:0:320700:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.318641:0:320700:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.318644:0:320700:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.318651:0:320700:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22dbcb8 00010000:00010000:2.0:1789431428.318655:0:320700:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.318664:0:320700:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.353150:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353164:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.353176:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353180:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353188:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353191:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353199:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353202:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353212:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353215:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353225:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353228:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353239:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353242:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353251:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353254:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353263:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353267:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.353462:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353468:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.353476:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353478:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353485:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353488:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353495:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353498:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353504:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353507:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353513:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353516:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353523:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353526:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353532:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353535:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353541:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353545:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.353683:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353689:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.353700:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353703:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353713:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353716:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353726:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353730:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353740:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353743:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.353752:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353755:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353764:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353767:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353775:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353778:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.353787:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.353791:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.354648:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354658:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.354669:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354672:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.354682:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354684:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.354691:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354694:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.354701:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354703:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.354708:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354710:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.354715:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354718:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.354725:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354727:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.354732:0:320708:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.354734:0:320708:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.401702:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.401722:0:320719:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.401730:0:320719:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.401758:0:320719:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.401777:0:320719:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.401792:0:320719:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.401799:0:320719:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.401806:0:320719:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.401810:0:320719:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.401817:0:320719:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c234bcb8 00010000:00010000:2.0:1789431428.401820:0:320719:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.401828:0:320719:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.410798:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410814:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.410828:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410863:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.410875:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410878:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.410885:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410888:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.410894:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410897:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.410904:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410908:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.410915:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410917:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.410924:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410926:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.410944:0:320719:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.410948:0:320719:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.484577:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.484597:0:320736:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.484611:0:320736:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.484638:0:320736:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.484657:0:320736:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.484678:0:320736:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.484686:0:320736:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.484694:0:320736:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.484698:0:320736:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.484703:0:320736:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2333cb8 00010000:00010000:2.0:1789431428.484710:0:320736:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.484718:0:320736:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.486896:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486904:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.486916:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486920:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.486928:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486942:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.486951:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486954:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.486960:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486963:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.486971:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486974:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.486981:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486984:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.486993:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.486996:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.487004:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487008:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.487199:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487205:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.487214:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487217:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.487224:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487227:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.487234:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487237:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.487244:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.487247:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489245:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489324:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489340:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489344:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489353:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489357:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489366:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489370:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.489508:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489512:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.489520:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489523:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489531:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489534:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489542:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489545:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489553:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489556:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489564:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489567:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489574:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489577:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489585:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489588:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489596:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489600:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.489723:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489728:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.489737:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489740:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489747:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489751:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489759:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489762:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489769:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489772:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.489779:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489782:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489790:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489793:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489800:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489803:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.489810:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.489814:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.491000:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491006:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.491014:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491017:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.491024:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491026:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.491033:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491035:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.491042:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491045:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.491055:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491059:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.491067:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491069:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.491077:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491079:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.491093:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.491097:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431428.493016:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493022:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.493032:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493036:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.493044:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493098:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.493697:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493704:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.493720:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493723:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.493732:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493735:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.493746:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493749:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.493758:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493761:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.493769:0:320736:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.493772:0:320736:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.512731:0:320740:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.512748:0:320740:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.512754:0:320740:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.512779:0:320740:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.512795:0:320740:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.512809:0:320740:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.512817:0:320740:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.512824:0:320740:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.512827:0:320740:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.512853:0:320740:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22abcb8 00010000:00010000:2.0:1789431428.512858:0:320740:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.513071:0:320740:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.520341:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.520360:0:320742:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.520367:0:320742:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.520393:0:320742:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.520412:0:320742:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.520426:0:320742:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.520435:0:320742:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.520442:0:320742:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.520446:0:320742:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.520453:0:320742:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22abcb8 00010000:00010000:2.0:1789431428.520456:0:320742:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.520465:0:320742:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.524314:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524329:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.524340:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524344:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.524352:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524355:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.524362:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524365:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.524372:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524374:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.524381:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524384:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.524391:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524393:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.524400:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524471:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.524485:0:320742:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.524489:0:320742:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.559570:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559584:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.559598:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559601:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.559610:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559612:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.559620:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559623:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.559631:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559634:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.559642:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559646:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.559653:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559656:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.559664:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559667:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.559677:0:320747:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.559680:0:320747:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.569244:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.569257:0:320750:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.569263:0:320750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.569287:0:320750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.569303:0:320750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.569317:0:320750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.569326:0:320750:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.569332:0:320750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.569335:0:320750:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.569342:0:320750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2353cb8 00010000:00010000:2.0:1789431428.569345:0:320750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.569353:0:320750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.570592:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570603:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.570614:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570618:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570627:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570631:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570640:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570644:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570653:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570657:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570666:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570669:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.570679:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570682:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.570694:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570697:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.570707:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570712:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.570926:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570945:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.570958:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570962:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570972:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570975:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570986:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.570990:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.570999:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571003:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571011:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571014:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571021:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571024:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571032:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571035:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571045:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571049:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.571194:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571199:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.571209:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571213:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571220:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571223:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571231:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571234:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571242:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571245:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571254:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571257:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571266:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571268:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571275:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571278:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571285:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571288:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.571423:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571428:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.571437:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571440:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571448:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571451:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571458:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571462:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571472:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571475:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571485:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571488:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571495:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571498:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571506:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571509:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571517:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571521:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.571630:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571635:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.571644:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571647:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571658:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571661:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571669:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571672:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571681:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571683:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571691:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571693:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571702:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571705:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571714:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571717:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.571724:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571728:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431428.571875:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571880:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.571889:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571892:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571901:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571904:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571913:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571916:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.571925:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.571928:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.574085:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.574091:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.574099:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.574102:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.574110:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.574113:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.574121:0:320750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.574124:0:320750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.639720:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639753:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.639771:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639776:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.639786:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639789:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.639797:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639800:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.639808:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639811:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.639818:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639821:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.639846:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639850:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.639862:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639866:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.639874:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.639878:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431428.640098:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.640104:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.642041:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642056:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642077:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642081:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642089:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642092:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642105:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642108:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642116:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642120:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642128:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642131:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642140:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642255:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642269:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642273:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431428.642446:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642452:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.642461:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642464:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642472:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642475:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642484:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642487:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642495:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642497:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642505:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642507:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642515:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642518:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642526:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642529:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642536:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642540:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431428.642669:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642673:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.642682:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642685:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642693:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642696:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642702:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642705:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642713:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642716:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642725:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642728:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642736:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642738:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642745:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642748:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642756:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642760:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431428.642914:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642918:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.642939:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642943:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642954:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642957:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642965:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642968:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642977:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642980:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.642988:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.642991:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.642998:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.643002:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.643009:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.643012:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.643020:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.643024:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431428.643135:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.643140:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.646315:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646322:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646337:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646340:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646347:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646350:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646356:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646359:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646365:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646368:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646374:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646376:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646382:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646384:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646391:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646395:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431428.646548:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646553:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.646561:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646564:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646572:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646574:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646581:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646584:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646592:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646594:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646601:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646603:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646609:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646612:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646620:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646622:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646629:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646632:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.646798:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646803:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.646813:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646816:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646825:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646828:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646857:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646861:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646870:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646873:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.646880:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646883:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646890:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646893:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646901:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646904:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.646911:0:320758:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.646915:0:320758:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431428.698473:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.698490:0:320767:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.698497:0:320767:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.698524:0:320767:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.698541:0:320767:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.698556:0:320767:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.698565:0:320767:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.698572:0:320767:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.698575:0:320767:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.698582:0:320767:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2383cb8 00010000:00010000:2.0:1789431428.698586:0:320767:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.698595:0:320767:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.700206:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700220:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.700231:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700234:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.700241:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700244:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.700250:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700253:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.700259:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700262:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.700268:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700271:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.700279:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700281:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.700289:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700293:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.700301:0:320767:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.700305:0:320767:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.718236:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718249:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.718261:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718265:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.718274:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718277:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.718287:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718291:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.718298:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718301:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.718309:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718312:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.718320:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718323:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.718331:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718334:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.718343:0:320770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.718346:0:320770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.743750:0:320778:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.743759:0:320778:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.743764:0:320778:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.743776:0:320778:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.743785:0:320778:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.743998:0:320778:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.744005:0:320778:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.744010:0:320778:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.744013:0:320778:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.744018:0:320778:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c235bcb8 00010000:00010000:2.0:1789431428.744021:0:320778:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.744028:0:320778:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.774242:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789431428.774631:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.774646:0:320785:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.774652:0:320785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.774678:0:320785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.774695:0:320785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.774711:0:320785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.774719:0:320785:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.774726:0:320785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.774730:0:320785:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.774736:0:320785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c23a3cb8 00010000:00010000:2.0:1789431428.774740:0:320785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.774748:0:320785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.776227:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776239:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.776253:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776257:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776273:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776276:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776284:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776287:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776295:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776298:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776306:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776310:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776318:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776321:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776329:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776332:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776339:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776342:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.776542:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776548:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.776563:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776566:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776578:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776580:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776588:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776591:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776599:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776601:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776609:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776612:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776620:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776622:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776629:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776635:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776641:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776644:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.776775:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776780:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.776789:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776792:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776803:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776807:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776818:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776821:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776859:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776862:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.776878:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776881:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776892:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776895:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776906:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776910:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.776922:0:320785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.776926:0:320785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.833696:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.833715:0:320792:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.833723:0:320792:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.833750:0:320792:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.833769:0:320792:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.833982:0:320792:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.833993:0:320792:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.834001:0:320792:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.834005:0:320792:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.834012:0:320792:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2bcbcb8 00010000:00010000:2.0:1789431428.834016:0:320792:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.834024:0:320792:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.836316:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836328:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.836342:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836346:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836355:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836358:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836368:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836371:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836378:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836381:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836387:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836391:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836398:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836401:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836411:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836414:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836421:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836424:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431428.836623:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836628:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.836646:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836650:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836666:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836670:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836686:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836689:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836709:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836713:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.836729:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836732:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836749:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836753:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836769:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836773:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.836790:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.836794:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431428.836995:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837001:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.837010:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837014:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837022:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837025:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837034:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837038:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837047:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837050:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837059:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837062:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.837069:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837071:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.837077:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837079:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.837085:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837088:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431428.837195:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837200:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.837209:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837212:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837219:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837222:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837228:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837231:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.837239:0:320792:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.837242:0:320792:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.865219:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.865314:0:320796:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.865322:0:320796:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.865351:0:320796:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.865374:0:320796:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.865392:0:320796:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.865401:0:320796:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.865409:0:320796:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.865413:0:320796:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.865419:0:320796:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2bd3cb8 00010000:00010000:2.0:1789431428.865423:0:320796:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.865432:0:320796:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.867828:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867871:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.867888:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867893:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.867902:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867905:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.867913:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867917:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.867924:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867938:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.867953:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867957:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.867966:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867969:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.867977:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867979:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.867986:0:320796:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.867989:0:320796:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431428.909439:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909455:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431428.909466:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909470:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.909479:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909481:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.909489:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909492:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.909500:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909503:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431428.909511:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909514:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.909522:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909525:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.909533:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909536:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431428.909544:0:320802:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431428.909547:0:320802:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:2.0:1789431428.924036:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913342848:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431428.924244:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789431428.924252:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913342976:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431428.924321:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913343104:192.168.203.128@tcp:400:kworker.0 00000080:00200000:2.0:1789431428.924651:0:320807:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431428.924665:0:320807:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431428.924673:0:320807:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431428.924698:0:320807:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431428.924716:0:320807:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.924730:0:320807:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431428.924739:0:320807:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431428.924745:0:320807:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431428.924749:0:320807:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431428.924756:0:320807:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2be3cb8 00010000:00010000:2.0:1789431428.924760:0:320807:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431428.924768:0:320807:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00080000:2.0:1789431428.929682:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2101180 x1876350913343104/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431444 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431428.929700:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913343104:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431428.929714:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913342976:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431428.939202:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913342848:192.168.203.128@tcp:400:kworker.0 00000080:00200000:2.0:1789431429.011491:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011506:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.011519:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011523:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.011532:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011535:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.011543:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011546:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.011555:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011558:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.011569:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011573:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.011581:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011584:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.011591:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011594:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.011603:0:320817:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.011607:0:320817:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.030389:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030400:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.030408:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030411:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030417:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030420:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030427:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030430:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030436:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030439:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030445:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030447:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030453:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030454:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030459:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030461:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030465:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030468:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.030562:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030567:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.030576:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030580:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030590:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030592:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030599:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030601:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030608:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030611:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.030617:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030620:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030627:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030630:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030638:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030640:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.030647:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.030650:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.031138:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031143:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.031153:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031156:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.031165:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031168:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.031177:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031179:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.031186:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031189:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.031197:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031200:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.031208:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031210:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.031216:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031219:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.031226:0:320821:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.031230:0:320821:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.098506:0:320831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.098525:0:320831:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.098532:0:320831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.098557:0:320831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.098575:0:320831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.098592:0:320831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.098600:0:320831:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.098607:0:320831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.098610:0:320831:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.098617:0:320831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c22abcb8 00010000:00010000:2.0:1789431429.098620:0:320831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.098629:0:320831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.109556:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.109574:0:320834:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.109581:0:320834:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.109608:0:320834:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.109626:0:320834:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.109642:0:320834:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.109651:0:320834:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.109657:0:320834:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.109661:0:320834:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.109740:0:320834:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c235bcb8 00010000:00010000:2.0:1789431429.109744:0:320834:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.109754:0:320834:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.112168:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112182:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.112197:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112202:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.112212:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112216:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.112226:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112230:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.112239:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112243:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.112253:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112257:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.112266:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112269:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.112277:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112281:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.112289:0:320834:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.112293:0:320834:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.143404:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.143420:0:320840:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.143427:0:320840:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.143452:0:320840:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.143469:0:320840:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.143485:0:320840:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.143493:0:320840:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.143501:0:320840:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.143504:0:320840:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.143512:0:320840:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2bfbcb8 00010000:00010000:2.0:1789431429.143516:0:320840:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.143526:0:320840:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.150464:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150487:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.150504:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150509:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.150517:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150519:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.150527:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150529:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.150537:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150540:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.150547:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150550:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.150558:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150560:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.150568:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150571:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.150579:0:320840:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.150583:0:320840:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.172917:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.172951:0:320848:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.172959:0:320848:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.172987:0:320848:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.173006:0:320848:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.173021:0:320848:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.173029:0:320848:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.173036:0:320848:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.173040:0:320848:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.173048:0:320848:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2c23cb8 00010000:00010000:2.0:1789431429.173051:0:320848:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.173060:0:320848:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.174461:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174474:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.174485:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174489:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174498:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174500:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174508:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174511:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174519:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174521:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174530:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174533:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174542:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174545:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174554:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174557:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174565:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174568:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431429.174767:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174772:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.174783:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174786:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174795:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174800:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174810:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174813:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174822:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174825:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.174859:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174863:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174874:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174877:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174885:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174888:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.174895:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.174899:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431429.175066:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175071:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.175082:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175085:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175094:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175097:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175105:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175108:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175117:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175120:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175129:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175132:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175141:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175144:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175152:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175155:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175164:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175168:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431429.175305:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175310:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.175320:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175323:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175332:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175335:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175342:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175345:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175354:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175357:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175366:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175369:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175375:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175379:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175387:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175390:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175397:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175401:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.175501:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175506:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.175513:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175516:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175522:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175525:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175532:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175534:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175540:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175543:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175549:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175552:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175560:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175563:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175570:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175572:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175579:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175582:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431429.175687:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175692:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.175699:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175702:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175708:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175711:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175719:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175722:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175729:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175731:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.175737:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175740:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175747:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175749:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175756:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175758:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.175764:0:320848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.175767:0:320848:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.245413:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.245432:0:320859:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.245440:0:320859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.245467:0:320859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.245484:0:320859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.245504:0:320859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.245511:0:320859:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.245517:0:320859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.245521:0:320859:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.245528:0:320859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2bf3cb8 00010000:00010000:2.0:1789431429.245532:0:320859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.245540:0:320859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.246890:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246904:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.246917:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246921:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.246944:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246948:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.246956:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246959:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.246966:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246968:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.246976:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246979:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.246987:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.246990:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.246999:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.247001:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.247008:0:320859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.247011:0:320859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.255043:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.255079:0:320861:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.255087:0:320861:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.255115:0:320861:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.255134:0:320861:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.255149:0:320861:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.255158:0:320861:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.255165:0:320861:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.255168:0:320861:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.255175:0:320861:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2bf3cb8 00010000:00010000:2.0:1789431429.255179:0:320861:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.255188:0:320861:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.256620:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256637:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.256651:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256655:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.256663:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256667:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.256676:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256679:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.256688:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256690:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.256698:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256701:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.256709:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256711:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.256719:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256722:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.256730:0:320861:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.256734:0:320861:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.296948:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.296968:0:320869:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.296976:0:320869:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.297000:0:320869:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.297018:0:320869:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.297032:0:320869:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.297040:0:320869:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.297046:0:320869:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.297049:0:320869:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.297055:0:320869:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2c83cb8 00010000:00010000:2.0:1789431429.297124:0:320869:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.297139:0:320869:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.298730:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298742:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.298753:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298757:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.298764:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298767:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.298774:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298777:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.298783:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298786:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.298793:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298796:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.298802:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298805:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.298812:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298815:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.298822:0:320869:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.298825:0:320869:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.307808:0:320872:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.307828:0:320872:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.364028:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.364046:0:320880:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.364054:0:320880:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.364081:0:320880:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.364099:0:320880:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.364115:0:320880:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.364123:0:320880:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.364131:0:320880:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.364134:0:320880:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.364141:0:320880:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2c93cb8 00010000:00010000:2.0:1789431429.364145:0:320880:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.364164:0:320880:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.366940:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.366956:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.366973:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.366977:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.366987:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.366990:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.366999:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367003:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.367010:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367013:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.367020:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367024:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.367031:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367035:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.367044:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367047:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.367054:0:320880:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.367058:0:320880:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.437281:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437296:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.437308:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437312:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.437322:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437325:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.437333:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437336:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.437344:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437346:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.437353:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437356:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.437364:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437368:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.437375:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437378:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.437387:0:320887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.437391:0:320887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.445047:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.445056:0:320891:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.445071:0:320891:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.445082:0:320891:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.445090:0:320891:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.445105:0:320891:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.445112:0:320891:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.445115:0:320891:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.445118:0:320891:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.445123:0:320891:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2cb3cb8 00010000:00010000:2.0:1789431429.445126:0:320891:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.445133:0:320891:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.447577:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447589:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.447600:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447604:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447614:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447618:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447626:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447629:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447637:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447640:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447647:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447651:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447658:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447660:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447668:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447671:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447679:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447682:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.447862:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447868:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.447876:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447879:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447886:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447889:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447896:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447899:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447907:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447910:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.447917:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447919:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447939:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447942:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447950:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447952:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.447960:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.447963:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.448169:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448174:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.448182:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448185:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.448193:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448195:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.448203:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448206:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.448216:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448220:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.448229:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448232:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.448240:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448243:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.448251:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448254:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.448262:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.448265:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.453638:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453655:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.453669:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453673:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.453680:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453684:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.453690:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453693:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.453700:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453703:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.453714:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453717:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.453726:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453729:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.453740:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453743:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.453751:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453754:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.453990:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.453994:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.454002:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454004:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454013:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454015:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454022:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454024:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454030:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454033:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454038:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454041:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454046:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454049:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454054:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454057:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454132:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454137:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.454245:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454250:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.454257:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454260:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454267:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454269:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454276:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454278:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454285:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454287:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454293:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454296:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454302:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454304:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454310:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454312:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454319:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454322:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.454432:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454436:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.454443:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454446:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454453:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454456:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454463:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454466:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454473:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454476:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454483:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454486:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454493:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454496:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454503:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454505:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454512:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454515:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.454668:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454673:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.454682:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454685:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454693:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454696:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454703:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454706:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454713:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454716:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.454723:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454725:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454732:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454735:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454742:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454744:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.454751:0:320891:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.454754:0:320891:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.485018:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.485040:0:320894:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.485048:0:320894:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.485154:0:320894:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.485174:0:320894:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.485189:0:320894:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.485198:0:320894:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.485205:0:320894:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.485210:0:320894:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.485217:0:320894:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2cbbcb8 00010000:00010000:2.0:1789431429.485221:0:320894:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.485230:0:320894:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.487558:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487571:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.487582:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487587:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.487595:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487598:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.487606:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487609:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.487618:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487621:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.487630:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487633:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.487641:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487644:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.487651:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487654:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.487662:0:320894:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.487667:0:320894:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.512554:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512569:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.512583:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512587:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512597:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512600:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512608:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512611:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512618:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512621:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512628:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512631:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.512640:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512642:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.512650:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512653:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.512662:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512666:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431429.512910:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512915:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.512939:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512943:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512953:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512957:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512965:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512968:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512977:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512980:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.512988:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.512991:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515018:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515032:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515047:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515052:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515063:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515069:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431429.515286:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515292:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.515303:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515306:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515316:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515319:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515328:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515331:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515339:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515342:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515350:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515354:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515363:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515366:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515376:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515379:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515389:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515393:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431429.515529:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515534:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.515546:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515550:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515559:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515563:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515573:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515576:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515585:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515589:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515597:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515600:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515609:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515612:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515620:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515623:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515631:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515635:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.515754:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515760:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.515768:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515771:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515780:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515783:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515794:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515797:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515806:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515809:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.515818:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515821:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515852:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515856:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515869:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515872:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.515881:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.515885:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431429.516045:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516050:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.516061:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516064:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.516075:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516078:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.516089:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516093:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.516103:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516106:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.516115:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516119:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.516127:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516130:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.516139:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516142:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.516150:0:320897:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.516154:0:320897:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.560385:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.560404:0:320907:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.560413:0:320907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.560442:0:320907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.560461:0:320907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.560479:0:320907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.560488:0:320907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.560494:0:320907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.560498:0:320907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.560506:0:320907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2cebcb8 00010000:00010000:2.0:1789431429.560510:0:320907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.560519:0:320907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.564250:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564268:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.564283:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564289:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.564302:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564305:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.564315:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564319:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.564328:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564331:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.564340:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564344:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.564352:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564354:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.564362:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564365:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.564373:0:320907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.564377:0:320907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.572419:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.572438:0:320910:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.572445:0:320910:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.572473:0:320910:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.572491:0:320910:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.572506:0:320910:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.572514:0:320910:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.572521:0:320910:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.572524:0:320910:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.572531:0:320910:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2c6bcb8 00010000:00010000:2.0:1789431429.572535:0:320910:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.572544:0:320910:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.582457:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582466:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.582476:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582479:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.582487:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582490:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.582497:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582500:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.582508:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582511:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.582520:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582523:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.582531:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582534:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.582542:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582545:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.582555:0:320910:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.582559:0:320910:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.589267:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589276:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.589290:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589295:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589304:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589307:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589315:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589318:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589326:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589330:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.589536:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589541:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.589551:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589554:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.589562:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589565:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.589571:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589574:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.589582:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589585:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.589593:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589596:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589604:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589607:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589615:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589618:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.589627:0:320912:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.589631:0:320912:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.621118:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.621128:0:320919:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.621132:0:320919:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.621145:0:320919:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.621156:0:320919:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.621166:0:320919:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.621173:0:320919:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.621177:0:320919:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.621180:0:320919:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.621185:0:320919:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d03cb8 00010000:00010000:2.0:1789431429.621188:0:320919:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.621196:0:320919:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.622401:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622413:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.622425:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622428:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622435:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622438:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622444:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622447:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622454:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622457:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622465:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622469:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622477:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622480:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622487:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622490:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622497:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622501:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.622637:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622642:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.622651:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622654:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622664:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622667:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622676:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622679:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622688:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622691:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622698:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622700:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622707:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622710:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622717:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622720:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.622727:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622731:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.622967:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622971:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.622979:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622982:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622989:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.622991:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.622999:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623001:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.623008:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623011:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.623017:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623020:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.623027:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623030:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.623038:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623041:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.623051:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.623055:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.625230:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625235:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.625243:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625246:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625253:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625256:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625266:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625268:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625276:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625279:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625287:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625290:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625298:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625301:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625309:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625311:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625320:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625324:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.625444:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625448:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.625524:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625528:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625538:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625541:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625549:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625552:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625560:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625563:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625572:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625575:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625584:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625587:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625594:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625597:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625672:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625677:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.625794:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625800:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.625808:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625811:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625819:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625822:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625851:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625855:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625864:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625868:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.625875:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625878:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625886:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625889:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625897:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625900:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.625907:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.625911:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.626034:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626040:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.626048:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626051:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626059:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626062:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626070:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626073:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626080:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626083:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626091:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626094:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626100:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626103:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626109:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626112:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626120:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626124:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.626254:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626259:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.626268:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626271:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626278:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626280:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626288:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626291:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626299:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626302:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.626310:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626313:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626321:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626324:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626332:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626335:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.626344:0:320919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.626347:0:320919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.647470:0:320922:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.647488:0:320922:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.647495:0:320922:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.647522:0:320922:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.647543:0:320922:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.647558:0:320922:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.647566:0:320922:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.647574:0:320922:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.647578:0:320922:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.647585:0:320922:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2ca3cb8 00010000:00010000:2.0:1789431429.647589:0:320922:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.647598:0:320922:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.659904:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.659916:0:320926:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.659933:0:320926:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.659956:0:320926:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.659972:0:320926:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.659987:0:320926:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.659994:0:320926:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.660001:0:320926:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.660005:0:320926:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.660010:0:320926:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d13cb8 00010000:00010000:2.0:1789431429.660014:0:320926:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.660022:0:320926:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.661359:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661372:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.661384:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661388:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661397:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661400:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661407:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661409:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661417:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661420:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661427:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661431:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661440:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661443:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661452:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661455:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661464:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661468:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.661623:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661629:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.661637:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661640:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661650:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661653:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661662:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661665:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661675:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661678:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661685:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661688:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661697:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661700:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661708:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661711:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661718:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661722:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.661889:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661895:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.661905:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661908:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661917:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661920:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661946:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661949:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661960:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661964:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.661975:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661978:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.661986:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.661990:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662000:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662003:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662016:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662021:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.662157:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662162:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.662172:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662175:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662184:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662187:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662195:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662199:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662208:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662211:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662221:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662224:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662234:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662238:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662247:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662251:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662260:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662264:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.662390:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662396:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.662406:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662410:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662419:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662423:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662432:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662435:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662444:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662447:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662457:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662460:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662470:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662473:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662485:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662488:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662499:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662503:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.662621:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662626:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.662634:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662637:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662644:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662647:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662655:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662659:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662667:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662670:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662678:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662681:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662688:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662691:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662699:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662702:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.662709:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662712:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.662860:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662865:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.662874:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662876:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662883:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662886:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662893:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662896:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662904:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662907:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.662915:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.662918:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.664879:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.664888:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.664902:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.664905:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.664915:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.664919:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.665149:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665154:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.665163:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665166:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.665177:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665180:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.665189:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665192:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.665201:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665205:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.665213:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665216:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.665223:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665226:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.665234:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665236:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.665245:0:320926:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.665249:0:320926:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.713250:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.713267:0:320933:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.713274:0:320933:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.713301:0:320933:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.713320:0:320933:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.713334:0:320933:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.713343:0:320933:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.713349:0:320933:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.713352:0:320933:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.713360:0:320933:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d23cb8 00010000:00010000:2.0:1789431429.713363:0:320933:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.713372:0:320933:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.714693:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714706:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.714718:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714722:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.714731:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714734:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.714741:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714744:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.714751:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714753:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.714761:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714766:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.714774:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714777:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.714786:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714789:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.714797:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.714801:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.714998:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715004:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.715013:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715016:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715025:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715028:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715035:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715037:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715044:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715047:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715053:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715056:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715062:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715064:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715071:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715073:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715081:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715084:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.715217:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715223:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.715231:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715234:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715240:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715243:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715251:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715254:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715260:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715263:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715271:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715274:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715281:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715284:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715293:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715297:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715306:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715310:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.715432:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715437:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.715444:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715446:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715453:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715455:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715462:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715465:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715472:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715475:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715482:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715484:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715491:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715493:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715500:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715503:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715510:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715513:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.715635:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715640:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.715651:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715653:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715660:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715663:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715669:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715671:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715677:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715680:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715685:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715688:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715694:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715696:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715702:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715705:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.715711:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715714:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.715827:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715857:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.715869:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715872:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715880:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715882:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715889:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715892:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715899:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715902:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.715911:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.715915:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.717494:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.717505:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.717524:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.717527:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.717534:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.717538:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.719778:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719789:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.719807:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719811:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.719820:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719823:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.719853:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719857:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.719869:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719872:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.719879:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719882:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.719890:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719893:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.719900:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719903:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.719912:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.719916:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.720112:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720117:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.720125:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720128:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.720134:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720136:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.720143:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720146:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.720154:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720157:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.720166:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720169:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.720177:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720180:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.720187:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720190:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.720196:0:320933:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.720199:0:320933:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.773478:0:320939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.773496:0:320939:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.773504:0:320939:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.773532:0:320939:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.773552:0:320939:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.773568:0:320939:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.773576:0:320939:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.773583:0:320939:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.773587:0:320939:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.773594:0:320939:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2ca3cb8 00010000:00010000:2.0:1789431429.773599:0:320939:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.773608:0:320939:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.783617:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.783637:0:320942:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.783644:0:320942:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.783685:0:320942:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.783704:0:320942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.783720:0:320942:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.783727:0:320942:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.783734:0:320942:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.783738:0:320942:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.783744:0:320942:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2ca3cb8 00010000:00010000:2.0:1789431429.783748:0:320942:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.783756:0:320942:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.785130:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785143:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.785156:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785160:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.785168:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785171:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.785179:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785182:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.785191:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785194:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.785201:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785205:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.785212:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785215:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.785222:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785225:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.785232:0:320942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.785235:0:320942:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.796013:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.796032:0:320947:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.796039:0:320947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.796062:0:320947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.796078:0:320947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.796092:0:320947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.796099:0:320947:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.796106:0:320947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.796109:0:320947:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.796115:0:320947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d2bcb8 00010000:00010000:2.0:1789431429.796119:0:320947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.796131:0:320947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.800245:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800257:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.800265:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800269:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800275:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800278:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800284:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800287:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800293:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800295:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800301:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800304:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800311:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800313:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800320:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800322:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800329:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800332:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.800455:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800459:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.800466:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800469:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800476:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800478:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800485:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800488:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800494:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800496:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.800503:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800506:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800512:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800514:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800520:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800523:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.800529:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.800532:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.800648:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801758:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.801786:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801792:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.801802:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801805:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.801813:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801815:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.801822:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801825:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.801846:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801850:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.801859:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801862:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.801869:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801872:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.801879:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.801882:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.802061:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802066:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.802076:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802079:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802086:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802089:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802096:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802099:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802107:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802109:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802117:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802120:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802128:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802131:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802140:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802143:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802151:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802155:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.802277:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802282:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.802292:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802294:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802302:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802304:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802312:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802315:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802322:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802325:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802333:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802336:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802344:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802347:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802355:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802358:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802367:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802370:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.802481:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802486:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.802496:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802499:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802508:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802511:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802520:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802523:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802531:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802534:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.802542:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802546:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802554:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802557:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802564:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802567:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.802575:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.802578:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.806022:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806033:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.806046:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806050:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806058:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806062:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806070:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806072:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806080:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806083:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806092:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806096:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806104:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806107:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806115:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806118:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806130:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806134:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.806335:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806340:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.806348:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806350:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806357:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806360:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806367:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806369:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806376:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806379:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.806386:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806389:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806395:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806397:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806405:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806407:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.806414:0:320947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.806418:0:320947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.849656:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.849666:0:320954:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.849673:0:320954:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.849744:0:320954:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.849762:0:320954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.849775:0:320954:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.849783:0:320954:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.849790:0:320954:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.849795:0:320954:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.849807:0:320954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d3bcb8 00010000:00010000:2.0:1789431429.849811:0:320954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.849821:0:320954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.852974:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.852988:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.853001:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853005:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853014:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853017:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853025:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853028:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853043:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853046:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853055:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853058:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853067:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853070:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853081:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853083:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853170:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853177:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431429.853411:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853417:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.853427:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853430:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853438:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853441:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853449:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853452:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853463:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853466:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853475:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853478:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853490:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853493:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853501:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853505:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.853514:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853519:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431429.853722:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853728:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.853739:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853742:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853750:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853754:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853763:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853765:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.853773:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.853776:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854083:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854149:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854167:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854171:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854180:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854183:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854190:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854300:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431429.854465:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854471:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.854485:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854489:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854505:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854509:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854520:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854524:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854607:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854613:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854623:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854626:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854634:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854637:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854647:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854649:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854659:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854663:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431429.854800:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854805:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.854815:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854819:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854829:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854856:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854874:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854879:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854897:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854901:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.854916:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854929:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854941:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854944:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854953:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854957:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.854965:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.854968:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.855158:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855163:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.855173:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855177:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855187:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855190:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855264:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855269:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855280:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855283:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855292:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855295:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855303:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855306:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855315:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855318:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855329:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855333:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431429.855460:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855465:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.855473:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855477:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855485:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855488:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855495:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855498:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855508:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855511:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855519:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855523:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855531:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855603:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855618:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855621:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855630:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855635:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.855808:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855813:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.855823:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855825:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855870:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855875:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855885:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855887:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855895:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855898:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.855905:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855909:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855917:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855931:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855944:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855947:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.855955:0:320954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.855959:0:320954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431429.898161:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431429.898182:0:320959:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431429.898190:0:320959:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431429.898215:0:320959:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431429.898234:0:320959:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.898249:0:320959:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.898257:0:320959:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431429.898265:0:320959:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431429.898269:0:320959:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431429.898276:0:320959:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2ca3cb8 00010000:00010000:2.0:1789431429.898281:0:320959:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431429.898291:0:320959:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431429.904219:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904227:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.904246:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904252:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.904274:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904279:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.904303:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904309:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.904331:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904336:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.904357:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904363:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.904381:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904387:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.904407:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904413:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.904429:0:320959:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.904434:0:320959:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.954025:0:320965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.954032:0:320965:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.954045:0:320965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.954048:0:320965:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.954057:0:320965:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.954061:0:320965:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431429.973226:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973240:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.973253:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973257:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973266:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973269:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973277:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973280:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973287:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973290:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973297:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973300:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973307:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973309:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973316:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973319:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973326:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973330:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431429.973521:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973526:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.973658:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973663:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973674:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973677:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973685:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973688:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973696:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973698:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973706:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973709:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973719:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973722:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973730:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973733:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973741:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973744:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431429.973910:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973916:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.973938:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973941:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973949:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973952:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973959:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973962:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973969:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973971:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.973978:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973981:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973989:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.973992:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.973999:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.974002:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.974010:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.974013:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431429.974151:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.974156:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.974165:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.974168:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975642:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975652:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975675:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975678:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975686:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975689:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975699:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975703:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.975710:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975712:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.975721:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975724:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.975730:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975734:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431429.975911:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975916:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.975936:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975940:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975948:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975950:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975957:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975960:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975968:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975971:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.975978:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975980:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.975987:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975989:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.975996:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.975998:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.976005:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976008:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431429.976130:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976135:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431429.976145:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976147:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.976155:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976158:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.976167:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976170:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.976177:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976180:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431429.976188:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976191:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.976200:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976203:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.976212:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976215:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431429.976224:0:320968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431429.976227:0:320968:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.020018:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.020040:0:320975:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.020047:0:320975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.020077:0:320975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.020560:0:320975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.020577:0:320975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.020587:0:320975:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.020595:0:320975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.020599:0:320975:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.020606:0:320975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d5bcb8 00010000:00010000:2.0:1789431430.020610:0:320975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.020620:0:320975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.023168:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023183:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.023195:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023201:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.023209:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023211:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.023219:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023222:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.023233:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023236:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.023245:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023248:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.023256:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023259:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.023266:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023269:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.023277:0:320975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.023280:0:320975:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.036631:0:320978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.036649:0:320978:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.036656:0:320978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.036681:0:320978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.036701:0:320978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.036717:0:320978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.036725:0:320978:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.036731:0:320978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.036734:0:320978:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.036740:0:320978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2cfbcb8 00010000:00010000:2.0:1789431430.036745:0:320978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.036752:0:320978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.049678:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.049690:0:320981:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.049697:0:320981:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.049722:0:320981:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.049738:0:320981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.049754:0:320981:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.049761:0:320981:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.049767:0:320981:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.049771:0:320981:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.049778:0:320981:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d63cb8 00010000:00010000:2.0:1789431430.049783:0:320981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.049792:0:320981:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.051193:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051205:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.051217:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051221:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.051230:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051232:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.051239:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051242:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.051249:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051252:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.051259:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051262:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.051269:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051272:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.051279:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051282:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.051289:0:320981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.051293:0:320981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.093321:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.093340:0:320990:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.093348:0:320990:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.093375:0:320990:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.093392:0:320990:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.093408:0:320990:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.093416:0:320990:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.093424:0:320990:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.093428:0:320990:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.093434:0:320990:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d7bcb8 00010000:00010000:2.0:1789431430.093438:0:320990:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.093446:0:320990:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.094690:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094701:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.094713:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094717:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.094727:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094730:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.094738:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094742:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.094749:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094752:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.094761:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094764:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.094771:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094774:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.094782:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094785:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.094792:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.094795:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431430.094995:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095000:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.095010:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095013:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095020:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095023:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095032:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095035:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095042:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095045:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095052:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095055:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095064:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095067:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095077:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095080:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095089:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095093:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431430.095231:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095236:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.095244:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095247:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095255:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095258:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095266:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095269:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095277:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095280:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095288:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095291:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095300:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095303:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095312:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095315:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095323:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095327:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431430.095453:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095459:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.095469:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095472:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095480:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095483:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095492:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095495:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095504:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095507:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095515:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095517:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095525:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095528:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095536:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095539:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095548:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095551:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431430.095673:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095678:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.095686:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095689:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095697:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095699:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095708:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095710:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095719:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095722:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.095732:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095735:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095744:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095747:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095757:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095760:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.095768:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.095771:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431430.095916:0:320990:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.097065:0:320990:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.124335:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.124344:0:320997:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.124351:0:320997:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.124377:0:320997:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.124394:0:320997:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.124409:0:320997:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.124418:0:320997:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.124424:0:320997:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.124428:0:320997:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.124435:0:320997:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d83cb8 00010000:00010000:2.0:1789431430.124438:0:320997:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.124446:0:320997:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.125747:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125759:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.125771:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125776:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.125784:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125787:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.125795:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125797:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.125805:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125807:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.125815:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125818:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.125827:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125852:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.125863:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125866:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.125873:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.125877:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431430.126030:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126035:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.126043:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126045:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126052:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126055:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126063:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126066:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126076:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126078:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126085:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126088:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126095:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126098:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126106:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126109:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126118:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126121:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431430.126246:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126252:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.126260:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126264:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126271:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126273:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126281:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126284:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126292:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126295:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126304:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126306:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126314:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126318:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126326:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126329:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126338:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126342:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431430.126471:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126476:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.126484:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126487:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126494:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126496:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126503:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126506:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126512:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126515:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126523:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126526:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126534:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126537:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126545:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126548:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126556:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126560:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431430.126677:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126683:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.126692:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126696:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126705:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126708:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126716:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126719:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126728:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126731:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.126741:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126744:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126753:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126755:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126762:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126765:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.126772:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.126775:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431430.126913:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130036:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.130069:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130076:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130086:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130090:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130100:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130103:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130111:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130114:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130122:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130125:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130134:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130136:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130144:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130146:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130159:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130163:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431430.130313:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130317:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.130326:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130329:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130338:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130341:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130349:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130351:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130359:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130361:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130369:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130371:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130381:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130384:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130392:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130395:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130403:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130407:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.130570:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130576:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.130586:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130590:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130598:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130601:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130610:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130613:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130621:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130624:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.130632:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130635:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130643:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130646:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130654:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130658:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.130666:0:320997:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.130670:0:320997:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431430.156974:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.156994:0:321003:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.157002:0:321003:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.157029:0:321003:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.157049:0:321003:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.157064:0:321003:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.157072:0:321003:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.157079:0:321003:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.157082:0:321003:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.157090:0:321003:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d3bcb8 00010000:00010000:2.0:1789431430.157093:0:321003:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.157101:0:321003:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.158413:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158427:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.158438:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158443:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158451:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158454:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158462:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158465:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158472:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158475:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158483:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158487:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158496:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158499:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158507:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158510:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158520:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158523:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431430.158712:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158718:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.158728:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158731:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158740:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158743:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158755:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158758:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158767:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158770:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158778:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158780:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158787:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158789:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158796:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158799:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.158806:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158809:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431430.158975:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158980:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.158989:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.158991:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.158998:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159000:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159006:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159009:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159016:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159018:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159025:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159027:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159034:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159037:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159045:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159047:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159054:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159057:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431430.159166:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159171:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.159179:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159181:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159189:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159191:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159198:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159200:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159206:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159209:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159216:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159219:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159225:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159228:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159233:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159235:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159242:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159245:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431430.159344:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159349:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.159357:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159360:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159366:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159369:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159376:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159379:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159386:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159389:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159396:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159399:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159406:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159410:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159417:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159419:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159427:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159430:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431430.159553:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159558:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.159566:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159569:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159576:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159579:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159586:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159589:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159598:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159600:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.159607:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159610:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159618:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159620:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159627:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159630:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.159636:0:321003:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.159639:0:321003:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.209337:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209352:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.209365:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209369:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.209378:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209381:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.209389:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209392:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.209400:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209404:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.209412:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209416:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.209424:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209427:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.209434:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209437:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.209445:0:321010:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.209449:0:321010:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.218428:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.218448:0:321013:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.218456:0:321013:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.218483:0:321013:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.218500:0:321013:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.218514:0:321013:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.218521:0:321013:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.218538:0:321013:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.218542:0:321013:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.218548:0:321013:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d93cb8 00010000:00010000:2.0:1789431430.218552:0:321013:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.218560:0:321013:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.219877:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219891:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.219903:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219907:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.219916:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219932:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.219943:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219947:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.219955:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219958:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.219966:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219970:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.219978:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219981:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.219989:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.219992:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.220001:0:321013:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.220005:0:321013:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.231646:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.231668:0:321017:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.231677:0:321017:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.231706:0:321017:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.231725:0:321017:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.231740:0:321017:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.231753:0:321017:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.231761:0:321017:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.231765:0:321017:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.231772:0:321017:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2da3cb8 00010000:00010000:2.0:1789431430.231776:0:321017:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.231784:0:321017:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.242806:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242820:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.242849:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242853:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.242862:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242864:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.242871:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242873:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.242881:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242883:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.242890:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242893:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.242899:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242902:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.242908:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242911:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.242927:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.242931:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431430.243102:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243107:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.243118:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243122:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243130:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243133:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243141:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243144:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243151:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243154:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243162:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243165:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243171:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243174:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243180:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243183:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243191:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243194:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431430.243327:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243333:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.243343:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243346:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243355:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243358:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243366:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243369:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243378:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243381:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243389:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243392:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243401:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243404:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243412:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243415:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243425:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243429:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431430.243554:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243559:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.243567:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243569:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243576:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243579:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243588:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243591:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243599:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243602:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243609:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243611:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243618:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243621:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243627:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243630:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243637:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243641:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431430.243753:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243758:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.243767:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243770:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243778:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243781:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243789:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243791:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243800:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243803:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.243811:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243815:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.243822:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.243825:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244673:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244676:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244682:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244684:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244692:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244695:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244701:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244704:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244710:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244712:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244718:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244720:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244727:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244731:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:3.0:1789431430.244878:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244883:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.244893:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244896:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244904:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244907:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244915:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244930:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244941:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244944:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.244951:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244954:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244962:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244965:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244973:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244976:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.244984:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.244988:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:3.0:1789431430.245108:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245113:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.245122:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245126:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245135:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245138:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245147:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245150:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245158:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245161:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245169:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245173:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245182:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245185:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245193:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245195:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245202:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245205:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431430.245320:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245325:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.245334:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245337:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245345:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245348:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245356:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245359:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245367:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245370:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245381:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245384:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245393:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245397:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245406:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245409:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245419:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245422:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431430.245538:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245543:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.245553:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245556:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245564:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245567:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245576:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245580:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245588:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245591:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245600:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245603:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245610:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245613:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245620:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245624:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245633:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245636:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.245803:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245809:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.245819:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245822:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245857:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245861:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245872:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245875:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245883:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245886:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.245895:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245897:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245906:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.245909:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.245916:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.246659:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.246669:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.246692:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.246697:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431430.247216:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247223:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.247233:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247236:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.247244:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247247:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.247255:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247258:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.247266:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247269:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.247277:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247280:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.247287:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247290:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.247299:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247302:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.247312:0:321017:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.247316:0:321017:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.248058:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.248100:0:321018:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.248105:0:321018:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431430.265027:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.265047:0:321021:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.265055:0:321021:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.265083:0:321021:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.265101:0:321021:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.265118:0:321021:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.265126:0:321021:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.265133:0:321021:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.265136:0:321021:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.265143:0:321021:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dabcb8 00010000:00010000:2.0:1789431430.265147:0:321021:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.265156:0:321021:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.267690:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267705:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.267717:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267721:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.267731:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267734:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.267743:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267746:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.267754:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267757:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.267765:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267769:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.267778:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267780:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.267786:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267788:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.267795:0:321021:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.267798:0:321021:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.275435:0:321024:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.275456:0:321024:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.275464:0:321024:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.275490:0:321024:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.275506:0:321024:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.275522:0:321024:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.275530:0:321024:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.275538:0:321024:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.275541:0:321024:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.275548:0:321024:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d0bcb8 00010000:00010000:3.0:1789431430.275552:0:321024:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.275561:0:321024:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.276143:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.276152:0:321025:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.276156:0:321025:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.276168:0:321025:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.276177:0:321025:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.276187:0:321025:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.276194:0:321025:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.276198:0:321025:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.276201:0:321025:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.276205:0:321025:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dbbcb8 00010000:00010000:3.0:1789431430.276208:0:321025:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.276215:0:321025:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.277453:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277464:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.277473:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277477:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277484:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277487:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277495:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277498:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277505:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277508:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277515:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277519:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277525:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277527:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277534:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277537:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277545:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277549:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:3.0:1789431430.277700:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277705:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.277713:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277715:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277723:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277726:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277733:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277736:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277742:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277745:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277751:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277753:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277761:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277763:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277770:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277772:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.277778:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277781:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:3.0:1789431430.277942:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277947:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.277956:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277959:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277966:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277969:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277976:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277978:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277985:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277988:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.277995:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.277998:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278007:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278010:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278018:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278021:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278028:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278032:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:3.0:1789431430.278155:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278161:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.278169:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278172:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278181:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278184:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278193:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278196:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278203:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278205:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278212:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278215:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278222:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278225:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278233:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278236:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278244:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278248:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:3.0:1789431430.278372:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278377:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.278386:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278389:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278397:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278400:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278407:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278410:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278417:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278419:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278428:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278430:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278439:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278442:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278451:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278454:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278462:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278465:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431430.278577:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278582:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.278592:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278595:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278604:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278606:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278615:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278618:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278625:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278629:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278636:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278639:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278646:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278648:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278654:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278657:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.278663:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278666:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431430.278780:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278785:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.278794:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278796:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278803:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278806:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278814:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278817:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.278825:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.278827:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.285369:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285383:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285396:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285399:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285409:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285411:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285420:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285423:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.285635:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285640:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.285650:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285654:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.285662:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285665:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.285672:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285675:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.285683:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285686:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.285695:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285698:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285706:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285710:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285718:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285721:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.285729:0:321025:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.285734:0:321025:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431430.313697:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.313717:0:321028:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.313724:0:321028:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.313753:0:321028:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.313771:0:321028:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.313786:0:321028:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.313794:0:321028:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.313802:0:321028:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.313806:0:321028:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.313812:0:321028:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d63cb8 00010000:00010000:3.0:1789431430.313816:0:321028:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.313826:0:321028:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.315145:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315157:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.315169:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315172:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.315181:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315184:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.315192:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315195:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.315204:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315206:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.315218:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315222:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.315232:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315235:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.315244:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315246:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.315254:0:321028:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.315257:0:321028:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.333755:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.333773:0:321032:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.333780:0:321032:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.333806:0:321032:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.333823:0:321032:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.333864:0:321032:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.333874:0:321032:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.333882:0:321032:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.333885:0:321032:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.333893:0:321032:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dc3cb8 00010000:00010000:2.0:1789431430.333897:0:321032:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.333906:0:321032:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.343410:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.343426:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.343438:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.343442:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.344440:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.344525:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.346161:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346176:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.346186:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346189:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.346198:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346201:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.346209:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346211:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.346218:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346221:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.346227:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.346230:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:3.0:1789431430.347296:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347306:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.347316:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347319:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347328:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347330:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347340:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347343:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347349:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347351:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347358:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347361:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347367:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347369:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347377:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347379:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347392:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347395:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:3.0:1789431430.347549:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347554:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.347564:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347567:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347573:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347576:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347582:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347585:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347591:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347594:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347600:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347603:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347610:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347612:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347619:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347622:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347629:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347632:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:3.0:1789431430.347764:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347768:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.347775:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347778:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347786:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347788:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347796:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347799:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347806:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347808:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.347815:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347818:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.347825:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.347827:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.348699:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348711:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.348725:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348729:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:3.0:1789431430.348909:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348914:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.348936:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348940:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.348949:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348952:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.348960:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348963:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.348971:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348975:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.348982:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348985:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.348995:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.348998:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349007:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349010:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349018:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349021:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431430.349152:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349158:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.349167:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349170:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349178:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349180:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349187:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349190:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349198:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349201:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349210:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349213:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349219:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349222:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349229:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349232:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349238:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349241:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431430.349339:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349344:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.349353:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349356:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349363:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349365:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349372:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349374:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349380:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349383:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349389:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349392:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349398:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349400:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349406:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349409:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349415:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349418:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.349562:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349567:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.349574:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349576:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349585:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349589:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349598:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349601:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349614:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349617:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.349626:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349629:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349638:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349640:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.349647:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.349650:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.352702:0:321032:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.352715:0:321032:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431430.369571:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.369589:0:321035:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.369597:0:321035:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.369622:0:321035:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.369639:0:321035:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.369653:0:321035:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.369661:0:321035:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.369669:0:321035:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.369673:0:321035:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.369680:0:321035:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2d9bcb8 00010000:00010000:3.0:1789431430.369684:0:321035:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.369697:0:321035:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.372127:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372143:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.372154:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372158:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.372166:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372168:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.372176:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372178:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.372184:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372187:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.372193:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372196:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.372205:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372208:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.372215:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372217:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.372224:0:321035:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.372227:0:321035:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.378180:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431430.378201:0:321037:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431430.378208:0:321037:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431430.378233:0:321037:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431430.378254:0:321037:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.378269:0:321037:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.378277:0:321037:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431430.378284:0:321037:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431430.378288:0:321037:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:2.0:1789431430.378294:0:321037:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dabcb8 00010000:00010000:2.0:1789431430.378299:0:321037:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431430.378307:0:321037:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431430.379601:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379612:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.379622:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379625:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.379634:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379637:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.379643:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379646:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.379653:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379655:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.379663:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379666:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.379674:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379677:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.379684:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379687:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.379695:0:321037:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.379699:0:321037:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.390144:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390159:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.390171:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390175:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390181:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390184:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390191:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390194:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390200:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390203:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390209:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390212:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390219:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390222:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390228:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390231:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390237:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390241:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431430.390358:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390363:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.390370:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390373:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390380:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390383:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390390:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390392:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390401:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390404:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390412:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390414:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390423:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390425:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390433:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390435:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390442:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390445:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431430.390551:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390556:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.390563:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390566:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390572:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390574:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390581:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390583:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390589:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390591:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390599:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390602:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390609:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390612:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390618:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390621:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390627:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390630:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431430.390730:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390735:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.390743:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390746:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390753:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390756:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390764:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390767:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390773:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390776:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.390783:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390785:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390793:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390795:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390805:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390808:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.390816:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.390820:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431430.391255:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391261:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.391270:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391273:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391280:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391283:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391291:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391294:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391302:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391305:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391313:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391316:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391323:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391325:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391332:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391334:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391341:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391344:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.391515:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391520:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.391528:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391531:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391537:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391540:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391546:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391549:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391556:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391560:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.391569:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391572:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391579:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391582:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391590:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391593:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.391603:0:321039:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.391607:0:321039:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431430.406362:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.406382:0:321042:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.406389:0:321042:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.406415:0:321042:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.406433:0:321042:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.406446:0:321042:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.406454:0:321042:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.406461:0:321042:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.406464:0:321042:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.406470:0:321042:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dd3cb8 00010000:00010000:3.0:1789431430.406474:0:321042:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.406483:0:321042:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.407716:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407727:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.407737:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407740:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.407748:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407751:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.407758:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407761:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.407767:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407770:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.407776:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407780:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.407786:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407788:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.407796:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407799:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.407807:0:321042:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.407810:0:321042:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.414755:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.414773:0:321045:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.414780:0:321045:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.414805:0:321045:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.414823:0:321045:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.414980:0:321045:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.414989:0:321045:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.414997:0:321045:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.415001:0:321045:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.415008:0:321045:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2dcbcb8 00010000:00010000:3.0:1789431430.415013:0:321045:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.415021:0:321045:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.416422:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416433:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.416443:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416447:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.416456:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416459:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.416466:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416468:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.416476:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416479:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.416488:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416491:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.416500:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416503:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.416512:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416515:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.416523:0:321045:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.416527:0:321045:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431430.436501:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436514:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431430.436525:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436529:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.436538:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436540:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.436549:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436551:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.436558:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436561:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431430.436568:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436571:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.436580:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436583:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.436590:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436593:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431430.436600:0:321051:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.436603:0:321051:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.442602:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.442620:0:321054:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.442627:0:321054:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.442651:0:321054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.442668:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.442684:0:321054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.442691:0:321054:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.442698:0:321054:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.442701:0:321054:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.442708:0:321054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2debcb8 00010000:00010000:3.0:1789431430.442711:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.442720:0:321054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.444077:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444090:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.444102:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444106:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.444114:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444117:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.444125:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444128:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.444137:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444140:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.444149:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444152:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.444160:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444163:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.444173:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444176:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.444184:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.444193:0:321054:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libacl.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a39500, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431430.444200:0:321054:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libacl.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431430.444219:0:321054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libacl.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431430.444277:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.444286:0:321054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.444297:0:321054:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.444309:0:321054:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913343360/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.444318:0:321054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.444340:0:321054:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343360:192.168.203.128@tcp:101:sed.0 00000100:00100000:3.0:1789431430.444639:0:321054:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.448828:0:321054:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343360:192.168.203.128@tcp:101:sed.0 00010000:00010000:3.0:1789431430.449280:0:321054:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449293:0:321054:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449300:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449308:0:321054:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449316:0:321054:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449322:0:321054:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.449326:0:321054:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449343:0:321054:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449351:0:321054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449358:0:321054:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.449364:0:321054:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.449370:0:321054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d951 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.449387:0:321054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913343360/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sed.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.449400:0:321054:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431430.449428:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.449440:0:321054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00010000:3.0:1789431430.449460:0:321054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2deb9a0 00010000:00010000:3.0:1789431430.449464:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.449473:0:321054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.449785:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449794:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.449806:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449810:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.449818:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449821:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.449828:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449859:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.449871:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449874:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.449883:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449887:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.449896:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449899:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.449909:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449913:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.449933:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.449945:0:321054:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libselinux.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c3985380, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431430.449951:0:321054:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libselinux.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431430.449970:0:321054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libselinux.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431430.450015:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.450023:0:321054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.450033:0:321054:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.450041:0:321054:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913343616/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.450050:0:321054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.450068:0:321054:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343616:192.168.203.128@tcp:101:sed.0 00000100:00100000:3.0:1789431430.450356:0:321054:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.456044:0:321054:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343616:192.168.203.128@tcp:101:sed.0 00010000:00010000:3.0:1789431430.456068:0:321054:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456082:0:321054:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456089:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456096:0:321054:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456104:0:321054:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456115:0:321054:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.456120:0:321054:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456127:0:321054:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456133:0:321054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456140:0:321054:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.456146:0:321054:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.456152:0:321054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27d958 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.456168:0:321054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913343616/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sed.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.456182:0:321054:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431430.456207:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.456218:0:321054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00010000:3.0:1789431430.456225:0:321054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2deb9a0 00010000:00010000:3.0:1789431430.456228:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.456240:0:321054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.456599:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456607:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.456618:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456622:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.456631:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456634:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.456643:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456646:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.456655:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456658:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.456666:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456670:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.456678:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456681:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.456705:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456709:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.456724:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.456729:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431430.457084:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457090:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.457099:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457102:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.457111:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457114:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.457122:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457125:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.457133:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457136:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.457145:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457149:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.457156:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457159:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.457167:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457169:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.457177:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.457186:0:321054:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libattr.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c4d13c80, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431430.457193:0:321054:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libattr.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431430.457213:0:321054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libattr.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431430.457260:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.457268:0:321054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.457277:0:321054:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.457284:0:321054:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913343872/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.457292:0:321054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.457313:0:321054:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343872:192.168.203.128@tcp:101:sed.0 00000100:00100000:3.0:1789431430.457484:0:321054:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.462000:0:321054:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913343872:192.168.203.128@tcp:101:sed.0 00010000:00010000:3.0:1789431430.462023:0:321054:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462037:0:321054:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462044:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462051:0:321054:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462059:0:321054:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462066:0:321054:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.462071:0:321054:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462077:0:321054:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462085:0:321054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462091:0:321054:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.462097:0:321054:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.462104:0:321054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27d95f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.462121:0:321054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913343872/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sed.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.462438:0:321054:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431430.462466:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.462478:0:321054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00010000:3.0:1789431430.462488:0:321054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2deb9a0 00010000:00010000:3.0:1789431430.462492:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.462499:0:321054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.463062:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463072:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.463082:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463086:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.463094:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463097:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.463105:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463108:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.463118:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463121:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.463328:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463333:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.463347:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463350:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.463361:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463364:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.463371:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.463381:0:321054:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre2-8.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2c46a80, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431430.463387:0:321054:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre2-8.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431430.463407:0:321054:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre2-8.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431430.463454:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.463463:0:321054:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.463474:0:321054:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.463483:0:321054:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913344128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.463492:0:321054:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.463512:0:321054:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913344128:192.168.203.128@tcp:101:sed.0 00000100:00100000:3.0:1789431430.463744:0:321054:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.467690:0:321054:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job sed:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321054:1876350913344128:192.168.203.128@tcp:101:sed.0 00010000:00010000:3.0:1789431430.467714:0:321054:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467726:0:321054:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467732:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467739:0:321054:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467748:0:321054:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467753:0:321054:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.467757:0:321054:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467764:0:321054:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467771:0:321054:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467777:0:321054:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.467782:0:321054:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.467789:0:321054:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d966 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321054 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.467806:0:321054:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913344128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sed.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.467820:0:321054:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431430.467988:0:321054:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.468003:0:321054:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00010000:3.0:1789431430.468012:0:321054:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2deb9a0 00010000:00010000:3.0:1789431430.468017:0:321054:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.468024:0:321054:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.468412:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468422:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.468434:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468438:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468448:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468451:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468463:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468466:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468475:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468478:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468487:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468490:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468497:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468500:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468507:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468509:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468517:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468521:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431430.468642:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468646:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431430.468651:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468653:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468658:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468659:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468665:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468666:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468672:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468673:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431430.468678:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468680:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468684:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468686:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468690:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468692:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431430.468696:0:321054:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.468698:0:321054:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431430.474654:0:319907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431430.474669:0:319907:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431430.474674:0:319907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431430.474696:0:319907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431430.474711:0:319907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.474722:0:319907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431430.474727:0:319907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431430.474732:0:319907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431430.474735:0:319907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27d88d 00000080:00010000:3.0:1789431430.474740:0:319907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d88d from it ffffbc42c2f5bcb8 00010000:00010000:3.0:1789431430.474742:0:319907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431430.474749:0:319907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2c00/0x1ffb2ee1bd27d88d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9740 expref: -99 pid: 317584 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:3.0:1789431430.678331:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d3af8000 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913351552:192.168.203.128@tcp:101:mkdir.0 00010000:00010000:3.0:1789431430.678354:0:321055:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678368:0:321055:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678375:0:321055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678382:0:321055:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678401:0:321055:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678408:0:321055:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c8c9f000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.678412:0:321055:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678419:0:321055:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678427:0:321055:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 5/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678435:0:321055:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.678441:0:321055:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.678447:0:321055:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9f000/0x1ffb2ee1bd27d9ac lrc: 0/0,0 mode: --/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.678471:0:321055:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9811d3af8000 x1876350913351552/t4294973748(4294973748) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431430.678484:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d3af8000 x1876350913351552/t4294973748(4294973748) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.678494:0:321055:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x66c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431430.678512:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66d:0x0], updating i_size 4096 00000080:00200000:3.0:1789431430.678516:0:321055:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x66d:0x0](ffff9811c4834b08): rc = 0 00000080:00010000:3.0:1789431430.678520:0:321055:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66d:0x0](ffff9811c4834b08) for lock 0x0 00000080:00200008:3.0:1789431430.678543:0:321055:0:(file.c:1090:ll_file_open()) START file 4b687f851f071e758fbbfa9992cc2960cce6382434d026b57948ce4fd1a6dc1243bc127c73873fb1b44a7a80e58492ff5de9aecce736dcf4982b7fbf94e03a4:[0x200000402:0x66d:0x0](ffff9811c4834b08), flags 704400 00000002:00100000:3.0:1789431430.678561:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d3af8000 x1876350913351552/t4294973748(4294973748) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.678608:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 44 00000080:00200000:3.0:1789431430.678630:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 101 00000080:00200008:3.0:1789431430.678645:0:321055:0:(file.c:479:ll_file_release()) START file 4b687f851f071e758fbbfa9992cc2960cce6382434d026b57948ce4fd1a6dc1243bc127c73873fb1b44a7a80e58492ff5de9aecce736dcf4982b7fbf94e03a4:[0x200000402:0x66d:0x0](ffff9811c4834b08), flags 704000 00000002:00100000:3.0:1789431430.678688:0:321055:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d3af8000 x1876350913351552/t4294973748(4294973748) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431430.678697:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431430.678703:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913351808/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431430.678721:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913351808:192.168.203.128@tcp:35:mkdir.0 00000100:00100000:3.0:1789431430.678790:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.682055:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913351808:192.168.203.128@tcp:35:mkdir.0 00000002:00100000:3.0:1789431430.682095:0:321055:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d3af8000 x1876350913351552/t4294973748(4294973748) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.682147:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.682168:0:321055:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, dir=[0x200000402:0x66d:0x0](ffff9811c4834b08), flags=1536 00000080:00200000:3.0:1789431430.682173:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.682178:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.682182:0:321055:0:(namei.c:2165:do_mkdir()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, dir=[0x200000402:0x66d:0x0](ffff9811c4834b08) 00000020:00100000:3.0:1789431430.682242:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431430.682250:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913352064/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 672/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431430.682284:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352064:192.168.203.128@tcp:36:mkdir.0 00000100:00100000:3.0:1789431430.682356:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.688157:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352064:192.168.203.128@tcp:36:mkdir.0 00000080:00200000:3.0:1789431430.688202:0:321055:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248) 00000080:00200000:3.0:1789431430.688208:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66e:0x0], updating i_size 4096 00000080:00200000:3.0:1789431430.688475:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.688497:0:321055:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, dir=[0x200000402:0x66d:0x0](ffff9811c4834b08), file ffff9811d8c54180, open_flags 38900, mode 0 opened 0 00000080:00200000:3.0:1789431430.688508:0:321055:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, dir=[0x200000402:0x66d:0x0](ffff9811c4834b08), intent=open 00000002:00010000:3.0:1789431430.688533:0:321055:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa,[0x0:0x0:0x0]) in obj [0x200000402:0x66d:0x0], intent: open flags 0704401 00010000:00010000:3.0:1789431430.688590:0:321055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.688600:0:321055:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.688610:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.688618:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7b80 x1876350913352320/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.688628:0:321055:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.688645:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352320:192.168.203.128@tcp:101:mkdir.0 00000100:00100000:3.0:1789431430.689233:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.693251:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352320:192.168.203.128@tcp:101:mkdir.0 00010000:00010000:3.0:1789431430.693287:0:321055:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693301:0:321055:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693308:0:321055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693319:0:321055:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693328:0:321055:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693334:0:321055:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.693339:0:321055:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693346:0:321055:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693352:0:321055:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693358:0:321055:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.693365:0:321055:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.693371:0:321055:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9b3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.693503:0:321055:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9811d1bc7b80 x1876350913352320/t4294973751(4294973751) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431430.693517:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc7b80 x1876350913352320/t4294973751(4294973751) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.693529:0:321055:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x66d:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431430.693548:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66e:0x0], updating i_size 4096 00000080:00200000:3.0:1789431430.693552:0:321055:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x66e:0x0](ffff9811c4823248): rc = 0 00000080:00010000:3.0:1789431430.693556:0:321055:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66e:0x0](ffff9811c4823248) for lock 0x0 00000080:00200008:3.0:1789431430.693582:0:321055:0:(file.c:1090:ll_file_open()) START file 6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa:[0x200000402:0x66e:0x0](ffff9811c4823248), flags 704400 00000002:00100000:3.0:1789431430.693602:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc7b80 x1876350913352320/t4294973751(4294973751) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.693646:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 44 00000080:00200000:3.0:1789431430.693666:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 101 00000080:00200008:3.0:1789431430.693681:0:321055:0:(file.c:479:ll_file_release()) START file 6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa:[0x200000402:0x66e:0x0](ffff9811c4823248), flags 704000 00000002:00100000:3.0:1789431430.693725:0:321055:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d1bc7b80 x1876350913352320/t4294973751(4294973751) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431430.693735:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431430.693740:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6a00 x1876350913352576/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431430.693761:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352576:192.168.203.128@tcp:35:mkdir.0 00000100:00100000:3.0:1789431430.694150:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.697117:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352576:192.168.203.128@tcp:35:mkdir.0 00000002:00100000:3.0:1789431430.697153:0:321055:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d1bc7b80 x1876350913352320/t4294973751(4294973751) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.697205:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.697226:0:321055:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, dir=[0x200000402:0x66e:0x0](ffff9811c4823248), flags=1536 00000080:00200000:3.0:1789431430.697231:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.697237:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.697242:0:321055:0:(namei.c:2165:do_mkdir()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, dir=[0x200000402:0x66e:0x0](ffff9811c4823248) 00000020:00100000:3.0:1789431430.697306:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431430.697313:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6a00 x1876350913352832/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 672/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431430.697334:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352832:192.168.203.128@tcp:36:mkdir.0 00000100:00100000:3.0:1789431430.698136:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.708669:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913352832:192.168.203.128@tcp:36:mkdir.0 00000080:00200000:3.0:1789431430.708721:0:321055:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348) 00000080:00200000:3.0:1789431430.708728:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66f:0x0], updating i_size 4096 00000080:00200000:3.0:1789431430.709380:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.709418:0:321055:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, dir=[0x200000402:0x66e:0x0](ffff9811c4823248), file ffff9811d8c54180, open_flags 38900, mode 0 opened 0 00000080:00200000:3.0:1789431430.709426:0:321055:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, dir=[0x200000402:0x66e:0x0](ffff9811c4823248), intent=open 00000002:00010000:3.0:1789431430.709449:0:321055:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603,[0x0:0x0:0x0]) in obj [0x200000402:0x66e:0x0], intent: open flags 0704401 00010000:00010000:3.0:1789431430.709511:0:321055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.709521:0:321055:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431430.709531:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431430.709540:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7b80 x1876350913353088/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431430.709550:0:321055:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431430.709571:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913353088:192.168.203.128@tcp:101:mkdir.0 00000100:00100000:3.0:1789431430.709745:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.714383:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913353088:192.168.203.128@tcp:101:mkdir.0 00010000:00010000:3.0:1789431430.714406:0:321055:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714420:0:321055:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714428:0:321055:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714436:0:321055:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714445:0:321055:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714451:0:321055:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431430.714455:0:321055:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714462:0:321055:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714470:0:321055:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 5/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714476:0:321055:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431430.714483:0:321055:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431430.714490:0:321055:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27d9ba lrc: 0/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431430.714515:0:321055:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9811d1bc7b80 x1876350913353088/t4294973754(4294973754) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431430.714528:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc7b80 x1876350913353088/t4294973754(4294973754) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.714539:0:321055:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x66e:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431430.714560:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66f:0x0], updating i_size 4096 00000080:00200000:3.0:1789431430.714564:0:321055:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x66f:0x0](ffff9811c4825348): rc = 0 00000080:00010000:3.0:1789431430.714567:0:321055:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) for lock 0x0 00000080:00200008:3.0:1789431430.714593:0:321055:0:(file.c:1090:ll_file_open()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), flags 704400 00000002:00100000:3.0:1789431430.714613:0:321055:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc7b80 x1876350913353088/t4294973754(4294973754) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.714661:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 44 00000080:00200000:3.0:1789431430.714682:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 101 00000080:00200008:3.0:1789431430.714696:0:321055:0:(file.c:479:ll_file_release()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), flags 704000 00000002:00100000:3.0:1789431430.714740:0:321055:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d1bc7b80 x1876350913353088/t4294973754(4294973754) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431430.714750:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431430.714754:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4380 x1876350913353344/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431430.714778:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913353344:192.168.203.128@tcp:35:mkdir.0 00000100:00100000:2.0:1789431430.715443:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00010000:00010000:3.0:1789431430.823625:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431430.956336:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d3afb100 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913359488:192.168.203.128@tcp:35:mkdir.0 00000002:00100000:3.0:1789431430.956370:0:321055:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d3af9180 x1876350913359232/t4294973778(4294973778) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431430.956422:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x677:0x0](ffff9811c48363c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431430.956442:0:321055:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626, dir=[0x200000402:0x677:0x0](ffff9811c48363c8), flags=1536 00000080:00200000:3.0:1789431430.956446:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x677:0x0](ffff9811c48363c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.956451:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x677:0x0](ffff9811c48363c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431430.956455:0:321055:0:(namei.c:2165:do_mkdir()) VFS Op:name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626, dir=[0x200000402:0x677:0x0](ffff9811c48363c8) 00000020:00100000:3.0:1789431430.956521:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431430.956528:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5880 x1876350913359744/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 672/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431430.956550:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913359744:192.168.203.128@tcp:36:mkdir.0 00000100:00100000:3.0:1789431430.956674:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431430.964075:0:321055:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913359744:192.168.203.128@tcp:36:mkdir.0 00000080:00200000:2.0:1789431430.964424:0:321055:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x678:0x0](ffff9811c4807448) 00000080:00200000:2.0:1789431430.964432:0:321055:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x678:0x0], updating i_size 4096 00000080:00200000:2.0:1789431430.964485:0:321055:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x677:0x0](ffff9811c48363c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431430.964512:0:321055:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626, dir=[0x200000402:0x677:0x0](ffff9811c48363c8), file ffff9811d1184c00, open_flags 38900, mode 0 opened 0 00000080:00200000:2.0:1789431430.964519:0:321055:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626, dir=[0x200000402:0x677:0x0](ffff9811c48363c8), intent=open 00000002:00010000:2.0:1789431430.964542:0:321055:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626,[0x0:0x0:0x0]) in obj [0x200000402:0x677:0x0], intent: open flags 0704401 00010000:00010000:2.0:1789431430.964604:0:321055:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27d9f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431430.964613:0:321055:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27d9f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431430.964623:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431430.964633:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100a80 x1876350913360000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431430.964643:0:321055:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27d9f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 321055 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431430.964662:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913360000:192.168.203.128@tcp:101:mkdir.0 00000100:00100000:2.0:1789431430.964868:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64d80 going to sleep for 11 seconds 00000080:00200008:2.0:1789431430.996066:0:321055:0:(file.c:479:ll_file_release()) START file 211430ca1a13832fa494baa06a12d1fc7a8df7325ac3c978207bedeb6f3da23405dcbad3e32decc61fd3d0999a9dfe524999ac7f2b841c17e32d6d0302d4ac9:[0x200000402:0x679:0x0](ffff9811c37a4b08), flags 704000 00000002:00100000:2.0:1789431430.996114:0:321055:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d3afb100 x1876350913360768/t4294973784(4294973784) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 768/608 e 0 to 0 dl 1789431446 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'mkdir.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1789431430.996125:0:321055:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1789431430.996130:0:321055:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913361024/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1789431430.996204:0:321055:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job mkdir:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321055:1876350913361024:192.168.203.128@tcp:35:mkdir.0 00000100:00100000:2.0:1789431430.996329:0:321055:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64300 going to sleep for 11 seconds 00000080:00200000:3.0:1789431431.240539:0:321058:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431431.240561:0:321058:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431431.240570:0:321058:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431431.240594:0:321058:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431431.240608:0:321058:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431431.240660:0:321058:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321058 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.240671:0:321058:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321058 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.240680:0:321058:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321058 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.240707:0:321058:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321058:1876350913367552:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431431.241768:0:321058:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000080:00200000:2.0:1789431431.250791:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.256808:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.256860:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.262712:0:321057:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431431.262718:0:321057:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431431.262742:0:321057:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431431.262799:0:321057:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.262807:0:321057:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.262814:0:321057:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.262857:0:321057:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321057:1876350913367936:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431431.263048:0:321057:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431431.271508:0:321057:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321057:1876350913367936:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431431.271533:0:321057:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271546:0:321057:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271553:0:321057:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271560:0:321057:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271569:0:321057:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271576:0:321057:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431431.271580:0:321057:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271588:0:321057:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271595:0:321057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271602:0:321057:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431431.271609:0:321057:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.271615:0:321057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da54 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321057 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431431.271634:0:321057:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2101180 x1876350913367936/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431431.271649:0:321057:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431431.271676:0:321057:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431431.271689:0:321057:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431431.271702:0:321057:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27da3f from it ffffbc42c2ddb970 00010000:00010000:2.0:1789431431.271707:0:321057:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431431.271715:0:321057:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431431.271789:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.271798:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.271810:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.271814:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.271822:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.271825:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.271899:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.280987:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.281019:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.281025:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.281037:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.281046:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.281056:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.288745:0:321057:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.288757:0:321057:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d8c15800, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431431.293439:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.293466:0:321057:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_0ba3f147646d99534ed07cacdce5a452, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431431.293471:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431431.293476:0:321057:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431431.293486:0:321057:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_0ba3f147646d99534ed07cacdce5a452, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8/b090b83cdb059b0b9a7e56cdef4a35f059b04549fe6b59fd502286ce65037e5771ad1fd0482a44a961f7d55801c9380eba1fc7f5aa533d938f39b01978aa860/50b14394f3f2b33cd114826042f3f933c046f519f19a416b931cd07ec63c5510a3f760eeb18b368ca6c7566b9de6e6fcb111d6ed5423be29fe3fbe75a601e33/08c7e6faa7867800cd92e6973789bfff53695097d6cbe4a6c21175248131f9c6792e40039da037b7032bf78ce0bc6f1f0a45597818ca210f35e3cb061d28fc7/bfb343f12339efbf3ea97e8acb32c9e0fe4d1decde068808b215f39510a76f76f9333a07edb5775677a59c7b3e76a5453b9a6f1af08e4ef5ee0f8d5f42afa34/40f9a8b7e37467ea37093694b69c5be7d51167d27fcd48ba9357a45fd8539fc3a73e74499cf3e314032af3a24b4e7776b9a5a50494d6406d918ad16cfc32fc2/8a486aa249fb863429f04e473b623e746e54aa5a48815a7b99fde9d3fcd62c20723aad7a2aabf90af2950f2a2585d74f7ce2ae6e7af2e563288a7132b0997ea/a98ac66c5772a05a4d1f99057ad9d806085dc6cced132ac36814e38eef11d827baa957089b45de8d7005414865e11f1bc42f89760e076cbeb15efadf9f96c27/e57e584b96a4aed132073d894e82def875f555cbd63166f47cd3ce461acd5c96e18fa76c68aeed5175ec07adf896f1688cd197c9dc2504c617efc0960b163cb/4b687f851f071e758fbbfa9992cc2960cce6382434d026b57948ce4fd1a6dc1243bc127c73873fb1b44a7a80e58492ff5de9aecce736dcf4982b7fbf94e03a4/6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa/8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603/9ccc22f15315299866b57621dbae0af11161de1f7c6bbc6fb46bb7d8b2ccfd5bfc37c6d45d89865761c1f9ab16a21efbce622aa8185bc5ff900a1ae78ca4d48/6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc/325e89ae1ddd43b3c4a79d9ae3139c81dc4e6b33e96d3f433fb45962b2b86838779d5df8e3f34003e3ddba4776607d38139cdfbdecae714aec87cb862a38933/08134030808a580b7c6cc1ab58737af9a39bebbf4d6459d96547e33a70b369cbea1500abd4005a5fcc52eb367beb9b95f07513aeeaec34ae7ec0f6101ad5a6d/93ac41c4fedd497d74116854b8d765b347ca79bfe5bd36945762d7641074d246ca8f6f7c78493f1d24041c6b0863c9a325ecc1bed88c55c16d7f8723166de30/0ef06b3fc6b6949e90955125c0b94ac9360da8d386ae2795a887277d74b5e1be2109278fafd798def216ebe9778aabb3abede435afbb173935214ee2a6bf4dc/f999b268f496b67bef31140050f50b268d83fa2409e152b1c977a2f64d4b6b6c2a8127f9790dd42561685eb3c58cfdd2e88364c4675e3541ef8427029f07182/bb66d1925fa45851890eac7ac4a81680930232d74d83efaf8fc49bc4f08c424fe568aa3d8ffd9aa59bb30f4a4a8f3f4a5450e7ed3b6c0e72ed5f9cc098861df/72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626/211430ca1a13832fa494baa06a12d1fc7a8df7325ac3c978207bedeb6f3da23405dcbad3e32decc61fd3d0999a9dfe524999ac7f2b841c17e32d6d0302d4ac9/04a0e70005c09b943f2dbc3e56e8d1b573301efc3114d5c8926b173bb4d4d149e36ed85a904ef8b12fa0386f9f59a5a2ce39751667849aed4a7d8ac724456d6/52e813962eff92c2a7c477dd4088122e77fc5765c1df41be52708306 00010000:00010000:2.0:1789431431.293537:0:321057:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431431.293564:0:321057:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431431.293639:0:321057:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431431.293648:0:321057:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431431.293651:0:321057:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27da3f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795a9fac expref: -99 pid: 321058 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:2.0:1789431431.293670:0:321057:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431431.293683:0:321057:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101180 x1876350913368576/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 4640/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431431.293709:0:321057:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321057:1876350913368576:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431431.294353:0:321057:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431431.301925:0:321057:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321057:1876350913368576:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431431.301979:0:321057:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x683:0x0](ffff9811c4800908) 00000080:00200000:2.0:1789431431.301985:0:321057:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x683:0x0], updating i_size 4095 00000080:00200000:2.0:1789431431.310763:0:321060:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431431.310782:0:321060:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431431.310790:0:321060:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431431.310811:0:321060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431431.310824:0:321060:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431431.311005:0:321060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.311016:0:321060:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.311026:0:321060:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.311056:0:321060:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321060:1876350913368704:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431431.311305:0:321060:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431431.315524:0:321060:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321060:1876350913368704:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431431.315550:0:321060:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315562:0:321060:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315573:0:321060:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315584:0:321060:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315596:0:321060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315605:0:321060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315612:0:321060:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431431.315623:0:321060:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2102d80 x1876350913368704/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431431.315643:0:321060:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315657:0:321060:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431431.315662:0:321060:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431431.315688:0:321060:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431431.315693:0:321060:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27da69 00000080:00010000:2.0:1789431431.315698:0:321060:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431431.315701:0:321060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27da69 from it ffffbc42c2dfbcb8 00010000:00010000:2.0:1789431431.315705:0:321060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.315712:0:321060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431431.317163:0:321060:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.317178:0:321060:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431431.317182:0:321060:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431431.317203:0:321060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431431.317250:0:321060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.317259:0:321060:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.317268:0:321060:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.317294:0:321060:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321060:1876350913368832:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431431.317364:0:321060:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431431.321574:0:321060:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321060:1876350913368832:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431431.321595:0:321060:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321607:0:321060:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321613:0:321060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321620:0:321060:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321628:0:321060:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321634:0:321060:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431431.321638:0:321060:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321645:0:321060:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321652:0:321060:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321659:0:321060:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431431.321665:0:321060:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.321671:0:321060:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27da70 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431431.321689:0:321060:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2102d80 x1876350913368832/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431431.321703:0:321060:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431431.321729:0:321060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431431.321741:0:321060:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431431.321750:0:321060:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27da69 from it ffffbc42c2dfb970 00010000:00010000:2.0:1789431431.321754:0:321060:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431431.321761:0:321060:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27da69 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa031 expref: -99 pid: 321060 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431431.322331:0:321060:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.322346:0:321060:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431431.322351:0:321060:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431431.322368:0:321060:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431431.322406:0:321060:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27da77 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.322414:0:321060:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27da77 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.322423:0:321060:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27da77 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321060 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.322441:0:321060:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321060:1876350913368960:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431431.322551:0:321060:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431431.373809:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373819:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.373865:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373870:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.373880:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373883:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.373889:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373892:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.373899:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373902:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.373919:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373923:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.373933:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373936:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.373945:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373947:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.373955:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.373959:0:321062:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.374385:0:321062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.374400:0:321062:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_0ba3f147646d99534ed07cacdce5a452, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:2.0:1789431431.374404:0:321062:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_0ba3f147646d99534ed07cacdce5a452, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431431.374426:0:321062:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_0ba3f147646d99534ed07cacdce5a452,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431431.374479:0:321062:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27da9a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.374491:0:321062:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27da9a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.374502:0:321062:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27da9a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321062 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.374526:0:321062:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321062:1876350913369984:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431431.374613:0:321062:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64000 going to sleep for 11 seconds 00000080:00200000:2.0:1789431431.402743:0:321065:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431431.402764:0:321065:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431431.402772:0:321065:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431431.402793:0:321065:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431431.402808:0:321065:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431431.403152:0:321065:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.403163:0:321065:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.403175:0:321065:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.403201:0:321065:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321065:1876350913370368:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431431.404485:0:321065:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a646c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431431.411451:0:321065:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.411471:0:321065:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431431.411476:0:321065:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431431.411499:0:321065:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431431.411555:0:321065:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27daa8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.411566:0:321065:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27daa8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.411576:0:321065:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27daa8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321065 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.411600:0:321065:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321065:1876350913370496:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431431.411687:0:321065:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64900 going to sleep for 11 seconds 00000080:00200000:2.0:1789431431.455971:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.455980:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.456001:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456007:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.456024:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456029:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.456049:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456055:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.456073:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456078:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.456094:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456097:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.456109:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456112:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.456125:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456128:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.456139:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456143:0:321067:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.456680:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.456697:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=16 00000080:00200000:2.0:1789431431.456703:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431431.456727:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431431.456787:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27dad2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.456799:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27dad2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431431.456808:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27dad2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431431.457033:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913371648:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431431.457172:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a640c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.729050:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d3afb100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374464:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.729077:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x679:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729099:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729109:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729119:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729130:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729143:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729155:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729162:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.729173:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d3afb100 x1876350913374464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.729185:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729194:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1658 (0 0) 00000080:00200000:3.0:1789431431.729200:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x679:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.729220:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67a:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.729223:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67a:0x0](ffff9811c4875348): rc = 0 00000080:00010000:3.0:1789431431.729228:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67a:0x0](ffff9811c4875348) for lock 0x1ffb2ee1bd27db6c 00000080:00010000:3.0:1789431431.729242:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67a:0x0](ffff9811c4875348) 00000080:00010000:3.0:1789431431.729245:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db6c from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.729248:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729257:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9ee00/0x1ffb2ee1bd27db6c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa3f0 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.729283:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67a:0x0](ffff9811c4875348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.729299:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=52e813962eff92c2a7c477dd4088122e77fc5765c1df41be527083060affe99db81b0ba22730e08c99ab63bee0b38a6544b6836819240b27c2a8fbb8651a59c, dir=[0x200000402:0x67a:0x0](ffff9811c4875348), flags=16 00000080:00200000:3.0:1789431431.729304:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=52e813962eff92c2a7c477dd4088122e77fc5765c1df41be527083060affe99db81b0ba22730e08c99ab63bee0b38a6544b6836819240b27c2a8fbb8651a59c, dir=[0x200000402:0x67a:0x0](ffff9811c4875348), intent=0 00000002:00010000:3.0:1789431431.729323:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 52e813962eff92c2a7c477dd4088122e77fc5765c1df41be527083060affe99db81b0ba22730e08c99ab63bee0b38a6544b6836819240b27c2a8fbb8651a59c,[0x0:0x0:0x0]) in obj [0x200000402:0x67a:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.729358:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67a:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729365:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.729373:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.729391:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374592:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.729464:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.740319:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374592:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.740345:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67a:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740369:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740378:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740389:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740405:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740418:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740426:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740434:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.740446:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913374592/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.740459:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740470:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1659 (0 0) 00000080:00200000:3.0:1789431431.740476:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x67a:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.740498:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67b:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.740501:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67b:0x0](ffff9811c4873248): rc = 0 00000080:00010000:3.0:1789431431.740507:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67b:0x0](ffff9811c4873248) for lock 0x1ffb2ee1bd27db73 00000080:00010000:3.0:1789431431.740520:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67b:0x0](ffff9811c4873248) 00000080:00010000:3.0:1789431431.740524:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db73 from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.740528:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740541:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27db73 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67b:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa40c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.740571:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67b:0x0](ffff9811c4873248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.740593:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=fe0697ea87551db53ffaff6491671e8000156d8523c2e57b215cbfac6dee0c3ca7783168eb8669306b25b0dbfbf990216ac3df2f4928a92af7791576cdf1a76, dir=[0x200000402:0x67b:0x0](ffff9811c4873248), flags=16 00000080:00200000:3.0:1789431431.740598:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=fe0697ea87551db53ffaff6491671e8000156d8523c2e57b215cbfac6dee0c3ca7783168eb8669306b25b0dbfbf990216ac3df2f4928a92af7791576cdf1a76, dir=[0x200000402:0x67b:0x0](ffff9811c4873248), intent=0 00000002:00010000:3.0:1789431431.740616:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fe0697ea87551db53ffaff6491671e8000156d8523c2e57b215cbfac6dee0c3ca7783168eb8669306b25b0dbfbf990216ac3df2f4928a92af7791576cdf1a76,[0x0:0x0:0x0]) in obj [0x200000402:0x67b:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.740654:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67b:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740661:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.740670:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.740689:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374720:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.740772:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.746523:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374720:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.746550:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746572:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746586:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746596:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746607:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746618:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746626:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746633:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.746643:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913374720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.746655:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746665:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1660 (0 0) 00000080:00200000:3.0:1789431431.746672:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x67b:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.746692:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67c:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.746696:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67c:0x0](ffff9811c4875b88): rc = 0 00000080:00010000:3.0:1789431431.746700:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67c:0x0](ffff9811c4875b88) for lock 0x1ffb2ee1bd27db7a 00000080:00010000:3.0:1789431431.746713:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67c:0x0](ffff9811c4875b88) 00000080:00010000:3.0:1789431431.746717:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db7a from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.746720:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746728:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27db7a lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67c:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa428 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.746774:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67c:0x0](ffff9811c4875b88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.746792:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=39d7881dd1133a402bbd78c86d421603d06c1d6f2ce24cfb460eaa8162dd1a57d62d6cf70979cea2b7ea33d91223d05cebb0a5d9e3938e627c033c813981d99, dir=[0x200000402:0x67c:0x0](ffff9811c4875b88), flags=16 00000080:00200000:3.0:1789431431.746797:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=39d7881dd1133a402bbd78c86d421603d06c1d6f2ce24cfb460eaa8162dd1a57d62d6cf70979cea2b7ea33d91223d05cebb0a5d9e3938e627c033c813981d99, dir=[0x200000402:0x67c:0x0](ffff9811c4875b88), intent=0 00000002:00010000:3.0:1789431431.746817:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 39d7881dd1133a402bbd78c86d421603d06c1d6f2ce24cfb460eaa8162dd1a57d62d6cf70979cea2b7ea33d91223d05cebb0a5d9e3938e627c033c813981d99,[0x0:0x0:0x0]) in obj [0x200000402:0x67c:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.746897:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67c:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746918:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.746928:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.746948:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374848:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.747021:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.752502:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374848:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.752529:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67c:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752551:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752559:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752569:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752580:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752593:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752601:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752609:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.752621:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913374848/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.752634:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752645:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1661 (0 0) 00000080:00200000:3.0:1789431431.752652:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x67c:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.752672:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67d:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.752675:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67d:0x0](ffff9811c48b5348): rc = 0 00000080:00010000:3.0:1789431431.752681:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67d:0x0](ffff9811c48b5348) for lock 0x1ffb2ee1bd27db81 00000080:00010000:3.0:1789431431.752694:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67d:0x0](ffff9811c48b5348) 00000080:00010000:3.0:1789431431.752697:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db81 from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.752701:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752710:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27db81 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67d:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa444 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.752738:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67d:0x0](ffff9811c48b5348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.752772:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=81e0616b088d78a1cf55e6f19c9c612488c31014f090245fd1bb293c84b53ae0d817d489851305a58020f71ffbfc0a5e784f0b30cfa10f3df77d848b0258e1b, dir=[0x200000402:0x67d:0x0](ffff9811c48b5348), flags=16 00000080:00200000:3.0:1789431431.752777:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=81e0616b088d78a1cf55e6f19c9c612488c31014f090245fd1bb293c84b53ae0d817d489851305a58020f71ffbfc0a5e784f0b30cfa10f3df77d848b0258e1b, dir=[0x200000402:0x67d:0x0](ffff9811c48b5348), intent=0 00000002:00010000:3.0:1789431431.752798:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 81e0616b088d78a1cf55e6f19c9c612488c31014f090245fd1bb293c84b53ae0d817d489851305a58020f71ffbfc0a5e784f0b30cfa10f3df77d848b0258e1b,[0x0:0x0:0x0]) in obj [0x200000402:0x67d:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.752867:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752874:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.752883:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.752902:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374976:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.752984:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.756993:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913374976:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.757020:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757041:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757050:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757061:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757072:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757084:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757093:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757100:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.757118:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913374976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.757131:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757141:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1662 (0 0) 00000080:00200000:3.0:1789431431.757147:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x67d:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.757168:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67e:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.757171:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67e:0x0](ffff9811c48b5b88): rc = 0 00000080:00010000:3.0:1789431431.757177:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67e:0x0](ffff9811c48b5b88) for lock 0x1ffb2ee1bd27db88 00000080:00010000:3.0:1789431431.757190:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67e:0x0](ffff9811c48b5b88) 00000080:00010000:3.0:1789431431.757193:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db88 from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.757197:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757205:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27db88 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa460 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.757245:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67e:0x0](ffff9811c48b5b88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.757264:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=675e51a3754010fbaaccfced84a5af7819431d98a46a5c121d2680f890d520daaa58b24676f5a3e1cf25ce507a0438ddccee4b7b71d60caa52a1ef4e3484fcd, dir=[0x200000402:0x67e:0x0](ffff9811c48b5b88), flags=16 00000080:00200000:3.0:1789431431.757269:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=675e51a3754010fbaaccfced84a5af7819431d98a46a5c121d2680f890d520daaa58b24676f5a3e1cf25ce507a0438ddccee4b7b71d60caa52a1ef4e3484fcd, dir=[0x200000402:0x67e:0x0](ffff9811c48b5b88), intent=0 00000002:00010000:3.0:1789431431.757290:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 675e51a3754010fbaaccfced84a5af7819431d98a46a5c121d2680f890d520daaa58b24676f5a3e1cf25ce507a0438ddccee4b7b71d60caa52a1ef4e3484fcd,[0x0:0x0:0x0]) in obj [0x200000402:0x67e:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.757327:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67e:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757336:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.757345:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.757364:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913375104:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.757443:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.770446:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913375104:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.770473:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67e:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770496:0:321067:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770505:0:321067:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770516:0:321067:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770528:0:321067:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770539:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770553:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770560:0:321067:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.770572:0:321067:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913375104/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.770583:0:321067:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770593:0:321067:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1663 (0 0) 00000080:00200000:3.0:1789431431.770599:0:321067:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x67e:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.770619:0:321067:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67f:0x0], updating i_size 4096 00000080:00200000:3.0:1789431431.770622:0:321067:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x67f:0x0](ffff9811c48b3248): rc = 0 00000080:00010000:3.0:1789431431.770628:0:321067:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67f:0x0](ffff9811c48b3248) for lock 0x1ffb2ee1bd27db8f 00000080:00010000:3.0:1789431431.770640:0:321067:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x67f:0x0](ffff9811c48b3248) 00000080:00010000:3.0:1789431431.770644:0:321067:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27db8f from it ffffbc42c2dfbab0 00010000:00010000:3.0:1789431431.770647:0:321067:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.770656:0:321067:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27db8f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x67f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa47c expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.770685:0:321067:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67f:0x0](ffff9811c48b3248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.770709:0:321067:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=4b6ff6360f1c496c2c46aba86ef2a5d1a2de8bc56bc4f4afa848998c4dd171c8026e84009eb345a4c680a2a26d6c74a882c17f3c81c84b0a1ad708ab3a08257, dir=[0x200000402:0x67f:0x0](ffff9811c48b3248), flags=16 00000080:00200000:3.0:1789431431.770714:0:321067:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=4b6ff6360f1c496c2c46aba86ef2a5d1a2de8bc56bc4f4afa848998c4dd171c8026e84009eb345a4c680a2a26d6c74a882c17f3c81c84b0a1ad708ab3a08257, dir=[0x200000402:0x67f:0x0](ffff9811c48b3248), intent=0 00000002:00010000:3.0:1789431431.770732:0:321067:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: 4b6ff6360f1c496c2c46aba86ef2a5d1a2de8bc56bc4f4afa848998c4dd171c8026e84009eb345a4c680a2a26d6c74a882c17f3c81c84b0a1ad708ab3a08257,[0x0:0x0:0x0]) in obj [0x200000402:0x67f:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431431.771448:0:321067:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27db96 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67f:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.771460:0:321067:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27db96 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.771469:0:321067:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27db96 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x67f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321067 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.771491:0:321067:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913375232:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.771630:0:321067:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c931c000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.782930:0:321067:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321067:1876350913375232:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.782957:0:321067:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27db96 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x67f:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa498 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1789431431.800010:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431431.800025:0:321070:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431431.800032:0:321070:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431431.800051:0:321070:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431431.800067:0:321070:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431431.800078:0:321070:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431431.800084:0:321070:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431431.800089:0:321070:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431431.800091:0:321070:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:2.0:1789431431.800107:0:321070:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c2e1bcb8 00010000:00010000:2.0:1789431431.800110:0:321070:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431431.800117:0:321070:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1789431431.801447:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801459:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.801471:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801474:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801483:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801485:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801494:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801497:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801505:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801508:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801516:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801519:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801528:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801531:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801540:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801543:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801551:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801554:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431431.801715:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801720:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.801729:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801732:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801739:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801743:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801751:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801754:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801761:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801764:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.801773:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801776:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801785:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801788:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801796:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801799:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.801809:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801813:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.801993:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.801999:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.802008:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802012:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.802019:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802022:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.802031:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802034:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.802042:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802045:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.802053:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802056:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.802064:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802068:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.802077:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802080:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.802503:0:321070:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.802509:0:321070:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431431.813159:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813168:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431431.813190:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813195:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431431.813210:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813214:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431431.813230:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813235:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431431.813252:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813257:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431431.813275:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813281:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431431.813298:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813303:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431431.813320:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813325:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431431.813343:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.813348:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431431.814614:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814626:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8, flags=16 00000080:00200000:3.0:1789431431.814633:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x664:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814637:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=b090b83cdb059b0b9a7e56cdef4a35f059b04549fe6b59fd502286ce65037e5771ad1fd0482a44a961f7d55801c9380eba1fc7f5aa533d938f39b01978aa860, flags=16 00000080:00200000:3.0:1789431431.814640:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x665:0x0](ffff9811c4870908), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814644:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=50b14394f3f2b33cd114826042f3f933c046f519f19a416b931cd07ec63c5510a3f760eeb18b368ca6c7566b9de6e6fcb111d6ed5423be29fe3fbe75a601e33, flags=16 00000080:00200000:3.0:1789431431.814647:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x666:0x0](ffff9811c48700c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814650:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=08c7e6faa7867800cd92e6973789bfff53695097d6cbe4a6c21175248131f9c6792e40039da037b7032bf78ce0bc6f1f0a45597818ca210f35e3cb061d28fc7, flags=16 00000080:00200000:3.0:1789431431.814653:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x667:0x0](ffff9811c4876c08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814657:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=bfb343f12339efbf3ea97e8acb32c9e0fe4d1decde068808b215f39510a76f76f9333a07edb5775677a59c7b3e76a5453b9a6f1af08e4ef5ee0f8d5f42afa34, flags=16 00000080:00200000:3.0:1789431431.814660:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x668:0x0](ffff9811c48300c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814663:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=40f9a8b7e37467ea37093694b69c5be7d51167d27fcd48ba9357a45fd8539fc3a73e74499cf3e314032af3a24b4e7776b9a5a50494d6406d918ad16cfc32fc2, flags=16 00000080:00200000:3.0:1789431431.814667:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x669:0x0](ffff9811c4831148), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814670:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=8a486aa249fb863429f04e473b623e746e54aa5a48815a7b99fde9d3fcd62c20723aad7a2aabf90af2950f2a2585d74f7ce2ae6e7af2e563288a7132b0997ea, flags=16 00000080:00200000:3.0:1789431431.814673:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66a:0x0](ffff9811c4835b88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814677:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=a98ac66c5772a05a4d1f99057ad9d806085dc6cced132ac36814e38eef11d827baa957089b45de8d7005414865e11f1bc42f89760e076cbeb15efadf9f96c27, flags=16 00000080:00200000:3.0:1789431431.814680:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66b:0x0](ffff9811c4836c08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814684:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=e57e584b96a4aed132073d894e82def875f555cbd63166f47cd3ce461acd5c96e18fa76c68aeed5175ec07adf896f1688cd197c9dc2504c617efc0960b163cb, flags=16 00000080:00200000:3.0:1789431431.814687:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66c:0x0](ffff9811c48321c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814690:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=4b687f851f071e758fbbfa9992cc2960cce6382434d026b57948ce4fd1a6dc1243bc127c73873fb1b44a7a80e58492ff5de9aecce736dcf4982b7fbf94e03a4, flags=16 00000080:00200000:3.0:1789431431.814693:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814697:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, flags=16 00000080:00200000:3.0:1789431431.814700:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814704:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, flags=16 00000080:00200000:3.0:1789431431.814707:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814710:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=9ccc22f15315299866b57621dbae0af11161de1f7c6bbc6fb46bb7d8b2ccfd5bfc37c6d45d89865761c1f9ab16a21efbce622aa8185bc5ff900a1ae78ca4d48, flags=16 00000080:00200000:3.0:1789431431.814713:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x670:0x0](ffff9811c4833248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814716:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc, flags=16 00000080:00200000:3.0:1789431431.814719:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814723:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=325e89ae1ddd43b3c4a79d9ae3139c81dc4e6b33e96d3f433fb45962b2b86838779d5df8e3f34003e3ddba4776607d38139cdfbdecae714aec87cb862a38933, flags=16 00000080:00200000:3.0:1789431431.814726:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x672:0x0](ffff9811c4833a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814729:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=08134030808a580b7c6cc1ab58737af9a39bebbf4d6459d96547e33a70b369cbea1500abd4005a5fcc52eb367beb9b95f07513aeeaec34ae7ec0f6101ad5a6d, flags=16 00000080:00200000:3.0:1789431431.814732:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x673:0x0](ffff9811c4831988), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814735:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=93ac41c4fedd497d74116854b8d765b347ca79bfe5bd36945762d7641074d246ca8f6f7c78493f1d24041c6b0863c9a325ecc1bed88c55c16d7f8723166de30, flags=16 00000080:00200000:3.0:1789431431.814738:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x674:0x0](ffff9811c48721c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814741:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=0ef06b3fc6b6949e90955125c0b94ac9360da8d386ae2795a887277d74b5e1be2109278fafd798def216ebe9778aabb3abede435afbb173935214ee2a6bf4dc, flags=16 00000080:00200000:3.0:1789431431.814745:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x675:0x0](ffff9811c4874b08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814748:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f999b268f496b67bef31140050f50b268d83fa2409e152b1c977a2f64d4b6b6c2a8127f9790dd42561685eb3c58cfdd2e88364c4675e3541ef8427029f07182, flags=16 00000080:00200000:3.0:1789431431.814752:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x676:0x0](ffff9811c48342c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814755:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=bb66d1925fa45851890eac7ac4a81680930232d74d83efaf8fc49bc4f08c424fe568aa3d8ffd9aa59bb30f4a4a8f3f4a5450e7ed3b6c0e72ed5f9cc098861df, flags=16 00000080:00200000:3.0:1789431431.814758:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x677:0x0](ffff9811c48363c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814761:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626, flags=16 00000080:00200000:3.0:1789431431.814765:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x678:0x0](ffff9811c4807448), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814768:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=211430ca1a13832fa494baa06a12d1fc7a8df7325ac3c978207bedeb6f3da23405dcbad3e32decc61fd3d0999a9dfe524999ac7f2b841c17e32d6d0302d4ac9, flags=16 00000080:00200000:3.0:1789431431.814771:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x679:0x0](ffff9811c37a4b08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814774:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=04a0e70005c09b943f2dbc3e56e8d1b573301efc3114d5c8926b173bb4d4d149e36ed85a904ef8b12fa0386f9f59a5a2ce39751667849aed4a7d8ac724456d6, flags=16 00000080:00200000:3.0:1789431431.814777:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67a:0x0](ffff9811c4875348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814781:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=52e813962eff92c2a7c477dd4088122e77fc5765c1df41be527083060affe99db81b0ba22730e08c99ab63bee0b38a6544b6836819240b27c2a8fbb8651a59c, flags=16 00000080:00200000:3.0:1789431431.814783:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67b:0x0](ffff9811c4873248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814787:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fe0697ea87551db53ffaff6491671e8000156d8523c2e57b215cbfac6dee0c3ca7783168eb8669306b25b0dbfbf990216ac3df2f4928a92af7791576cdf1a76, flags=16 00000080:00200000:3.0:1789431431.814790:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67c:0x0](ffff9811c4875b88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814793:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=39d7881dd1133a402bbd78c86d421603d06c1d6f2ce24cfb460eaa8162dd1a57d62d6cf70979cea2b7ea33d91223d05cebb0a5d9e3938e627c033c813981d99, flags=16 00000080:00200000:3.0:1789431431.814796:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67d:0x0](ffff9811c48b5348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814800:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=81e0616b088d78a1cf55e6f19c9c612488c31014f090245fd1bb293c84b53ae0d817d489851305a58020f71ffbfc0a5e784f0b30cfa10f3df77d848b0258e1b, flags=16 00000080:00200000:3.0:1789431431.814802:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67e:0x0](ffff9811c48b5b88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814806:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=675e51a3754010fbaaccfced84a5af7819431d98a46a5c121d2680f890d520daaa58b24676f5a3e1cf25ce507a0438ddccee4b7b71d60caa52a1ef4e3484fcd, flags=16 00000080:00200000:3.0:1789431431.814809:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x67f:0x0](ffff9811c48b3248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814813:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=4b6ff6360f1c496c2c46aba86ef2a5d1a2de8bc56bc4f4afa848998c4dd171c8026e84009eb345a4c680a2a26d6c74a882c17f3c81c84b0a1ad708ab3a08257, flags=16 00000080:00200000:3.0:1789431431.814816:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x680:0x0](ffff9811c48742c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814819:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=201f658d8c4a8aef6d19b891516cc58b465515e77cd6326f79cdf686777dff7c31ab0d11b314bb14c04b9e8d4d002716ae8073be963932cab00c8a2189bb17b, flags=16 00000080:00200000:3.0:1789431431.814822:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x681:0x0](ffff9811c4871988), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814826:0:321072:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=2e9c93aaa2d4b22d46d8af6946fcb63b73a0905736d9c0b9bb17f26a122eb00ace26edb5925a0941d82c03c35f21fefb49d5ded082c4e8bc62daa3fb3a886f9, flags=16 00000080:00200000:3.0:1789431431.814829:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x682:0x0](ffff9811c4877448), inode mode 41ed mask 1 00000080:00200000:3.0:1789431431.814861:0:321072:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=c19a190568f4125c93f0c156816843107c3c799dadd968733a4a9606d44a681e5929e3fa63a4fb0d7d643efed4efa9f22ad3bd48e0d5313031548b4ee14a234, dir=[0x200000402:0x682:0x0](ffff9811c4877448), flags=0 00000080:00200000:3.0:1789431431.814866:0:321072:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=c19a190568f4125c93f0c156816843107c3c799dadd968733a4a9606d44a681e5929e3fa63a4fb0d7d643efed4efa9f22ad3bd48e0d5313031548b4ee14a234, dir=[0x200000402:0x682:0x0](ffff9811c4877448), intent=getattr 00000002:00010000:3.0:1789431431.814888:0:321072:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: c19a190568f4125c93f0c156816843107c3c799dadd968733a4a9606d44a681e5929e3fa63a4fb0d7d643efed4efa9f22ad3bd48e0d5313031548b4ee14a234,[0x0:0x0:0x0]) in obj [0x200000402:0x682:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431431.814992:0:321072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x682:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 321072 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.815002:0:321072:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x682:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321072 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.815011:0:321072:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x682:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 321072 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431431.815034:0:321072:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321072:1876350913375872:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431431.815113:0:321072:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.824745:0:321072:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321072:1876350913375872:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431431.824770:0:321072:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: --/CR res: [0x200000402:0x682:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824797:0:321072:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: --/PR res: [0x200000402:0x684:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824806:0:321072:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: --/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824817:0:321072:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824828:0:321072:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824885:0:321072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824893:0:321072:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.824901:0:321072:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431431.825063:0:321072:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4000 x1876350913375872/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 696/584 e 0 to 0 dl 1789431447 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431431.825078:0:321072:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.825093:0:321072:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1668 (0 0) 00000080:00200000:3.0:1789431431.825099:0:321072:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x682:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431431.825121:0:321072:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x684:0x0], updating i_size 42 00000080:00200000:3.0:1789431431.825125:0:321072:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x684:0x0](ffff9811c48763c8): rc = 0 00000080:00010000:3.0:1789431431.825130:0:321072:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x684:0x0](ffff9811c48763c8) for lock 0x1ffb2ee1bd27dbab 00000080:00010000:3.0:1789431431.825138:0:321072:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x684:0x0](ffff9811c48763c8) 00000080:00010000:3.0:1789431431.825140:0:321072:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dbab from it ffffbc42c2e0be00 00010000:00010000:3.0:1789431431.825144:0:321072:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431431.825151:0:321072:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431431.825177:0:321072:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x682:0x0](ffff9811c4877448), inode mode 41ed mask 3 00000080:00200000:3.0:1789431431.825184:0:321072:0:(namei.c:2344:ll_unlink()) VFS Op:name=c19a190568f4125c93f0c156816843107c3c799dadd968733a4a9606d44a681e5929e3fa63a4fb0d7d643efed4efa9f22ad3bd48e0d5313031548b4ee14a234, dir=[0x200000402:0x682:0x0](ffff9811c4877448) 00010000:00010000:3.0:1789431431.825211:0:321072:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431431.825249:0:321072:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 1/0,0 mode: --/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431431.825256:0:321072:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431431.825259:0:321072:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dbab lrc: 0/0,0 mode: --/PR res: [0x200000402:0x684:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa4fa expref: -99 pid: 321072 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1789431431.825281:0:321072:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431431.825289:0:321072:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4000 x1876350913376128/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 608/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431431.825312:0:321072:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321072:1876350913376128:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431431.825381:0:321072:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431431.834298:0:321072:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321072:1876350913376128:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431431.834350:0:321072:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x684:0x0](ffff9811c48763c8) 00000080:00200000:3.0:1789431431.845182:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431431.845198:0:321075:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431431.845205:0:321075:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431431.845225:0:321075:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431431.845243:0:321075:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431431.845258:0:321075:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1789431431.845267:0:321075:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431431.845274:0:321075:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431431.845277:0:321075:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:3.0:1789431431.845285:0:321075:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c0dd3cb8 00010000:00010000:3.0:1789431431.845289:0:321075:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431431.845307:0:321075:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1789431431.846323:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789431431.851633:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851648:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.851661:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851666:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.851678:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851682:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.851689:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851692:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.851698:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851701:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.851709:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851713:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.851722:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851725:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.851733:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851736:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.851742:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.851745:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431431.852329:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852337:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.852344:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852347:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852355:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852357:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852364:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852366:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852417:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852420:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852427:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852430:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852438:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852441:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852448:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852451:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852458:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852461:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.852597:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852602:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.852610:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852613:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852620:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852623:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852631:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852634:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852641:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852643:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.852650:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852653:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852661:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852663:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852670:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852673:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.852679:0:321075:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.852683:0:321075:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431431.867493:0:321078:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431431.867508:0:321078:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431431.867516:0:321078:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431431.867535:0:321078:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431431.867555:0:321078:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1789431431.867581:0:321078:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1789431431.867591:0:321078:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431431.867598:0:321078:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431431.867602:0:321078:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:3.0:1789431431.867609:0:321078:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c2debcb8 00010000:00010000:3.0:1789431431.867613:0:321078:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1789431431.867622:0:321078:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1789431431.881170:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431431.881187:0:321080:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431431.881194:0:321080:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431431.881213:0:321080:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431431.881231:0:321080:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1789431431.881252:0:321080:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1789431431.881261:0:321080:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431431.881268:0:321080:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431431.881272:0:321080:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:2.0:1789431431.881279:0:321080:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c2e2bcb8 00010000:00010000:2.0:1789431431.881283:0:321080:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1789431431.881293:0:321080:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1789431431.882671:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882685:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.882697:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882701:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.882711:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882713:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.882721:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882724:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.882730:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882732:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.882739:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882742:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.882748:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882751:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.882761:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882764:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.882771:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882775:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431431.882991:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.882996:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.883003:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883006:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883014:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883017:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883023:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883026:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883033:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883035:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883041:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883044:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883051:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883054:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883061:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883063:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883069:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883073:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.883191:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883195:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.883203:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883206:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883215:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883217:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883224:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883227:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883234:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883238:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.883245:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883248:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883255:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883258:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883267:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883270:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.883278:0:321080:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.883281:0:321080:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431431.888601:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431431.888621:0:321083:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431431.888629:0:321083:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431431.888651:0:321083:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431431.888671:0:321083:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431431.888692:0:321083:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431431.888701:0:321083:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431431.888708:0:321083:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431431.888711:0:321083:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:3.0:1789431431.888717:0:321083:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c2e0bcb8 00010000:00010000:3.0:1789431431.888720:0:321083:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431431.888727:0:321083:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431431.890445:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890459:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431431.890474:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890479:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.890492:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890496:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.890507:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890510:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.890522:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890526:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431431.890541:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890545:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.890560:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890564:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.890582:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890586:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431431.890602:0:321083:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431431.890606:0:321083:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431431.920251:0:319907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431431.920268:0:319907:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431431.920273:0:319907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431431.920291:0:319907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431431.920306:0:319907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431431.920318:0:319907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1789431431.920326:0:319907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431431.920332:0:319907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431431.920336:0:319907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27daa1 00000080:00010000:2.0:1789431431.920343:0:319907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27daa1 from it ffffbc42c2f5bcb8 00010000:00010000:2.0:1789431431.920346:0:319907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431431.920355:0:319907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1789431432.150663:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x667:0x0](ffff9811c4876c08),name=08c7e6faa7867800cd92e6973789bfff53695097d6cbe4a6c21175248131f9c6792e40039da037b7032bf78ce0bc6f1f0a45597818ca210f35e3cb061d28fc7 00000002:00010000:2.0:1789431432.150695:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x667:0x0]) in obj [0x200000402:0x666:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431432.150709:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1639 (0 0) 00010000:00010000:2.0:1789431432.150757:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27dbf1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x666:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.150767:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27dbf1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x666:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.150777:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27dbf1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x666:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431432.150805:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913379328:192.168.203.128@tcp:101:rm.0 00000100:00100000:2.0:1789431432.150933:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64900 going to sleep for 11 seconds 00000080:00200000:3.0:1789431432.409516:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248),name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa 00000002:00010000:3.0:1789431432.409539:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66e:0x0]) in obj [0x200000402:0x66d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.409550:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1646 (0 0) 00010000:00010000:3.0:1789431432.409596:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dc84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.409606:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dc84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.409615:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dc84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66d:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.409637:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913386496:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.409862:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000080:00200000:3.0:1789431432.447449:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348),name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603 00000002:00010000:3.0:1789431432.447479:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66e:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.447492:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1647 (0 0) 00010000:00010000:3.0:1789431432.447534:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.447541:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.447548:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.447566:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387520:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.447637:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.451616:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387520:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.451642:0:321086:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451661:0:321086:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451669:0:321086:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451680:0:321086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451697:0:321086:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451709:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451717:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451725:0:321086:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.451737:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4700 x1876350913387520/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431448 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431432.451749:0:321086:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.451772:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1647 (0 0) 00000080:00200000:3.0:1789431432.451779:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x66e:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.451798:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66f:0x0], updating i_size 4096 00000080:00010000:3.0:1789431432.451803:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) for lock 0x1ffb2ee1bd27dc99 00000080:00010000:3.0:1789431432.451813:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) 00000080:00010000:3.0:1789431432.451816:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dc99 from it ffffbc42c2dbbcb8 00010000:00010000:3.0:1789431432.451819:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.452348:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431432.452410:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.452420:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, flags=258 00000080:00200000:3.0:1789431432.452438:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 44 00000080:00200008:3.0:1789431432.452446:0:321086:0:(file.c:1090:ll_file_open()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), flags 704400 00000002:00010000:3.0:1789431432.452469:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66e:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1789431432.452519:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.452527:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431432.452537:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431432.452545:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4700 x1876350913387776/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431432.452555:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.452574:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387776:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.452742:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.456735:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387776:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.456758:0:321086:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456882:0:321086:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456889:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456896:0:321086:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456916:0:321086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456923:0:321086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431432.456927:0:321086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456935:0:321086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456943:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456949:0:321086:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431432.456955:0:321086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.456962:0:321086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.456980:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9811d1bc4700 x1876350913387776/t4294973848(4294973848) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431432.456994:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4700 x1876350913387776/t4294973848(4294973848) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431432.457007:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x66e:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.457027:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66f:0x0], updating i_size 4096 00000002:00100000:3.0:1789431432.457047:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4700 x1876350913387776/t4294973848(4294973848) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1789431432.457105:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431432.457240:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348),name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603 00000002:00010000:3.0:1789431432.457261:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66e:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.457275:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431432.457286:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431432.457294:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x66e:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431432.457299:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) 00000080:00010000:3.0:1789431432.457303:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) for lock 0x1ffb2ee1bd27dc99 00000080:00010000:3.0:1789431432.457308:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dc99 from it ffffbc42c2dbbcd8 00010000:00010000:3.0:1789431432.457311:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431432.457319:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431432.457363:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348) pos/size0/4096 32bit_api 0 00000002:00010000:3.0:1789431432.457375:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66f:0x0], intent: readdir flags 0 00010000:00010000:3.0:1789431432.457381:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431432.457386:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1789431432.458022:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387904:192.168.203.128@tcp:37:rm.0 00000100:00100000:3.0:1789431432.458096:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.465096:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913387904:192.168.203.128@tcp:37:rm.0 00010000:00010000:3.0:1789431432.465965:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431432.465981:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1789431432.466034:0:321086:0:(file.c:479:ll_file_release()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), flags 704000 00000002:00100000:3.0:1789431432.466092:0:321086:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d1bc4700 x1876350913387776/t4294973848(4294973848) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431432.466103:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431432.466108:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913388160/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431432.466130:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913388160:192.168.203.128@tcp:35:rm.0 00000100:00100000:3.0:1789431432.466209:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.469693:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913388160:192.168.203.128@tcp:35:rm.0 00000002:00100000:3.0:1789431432.469729:0:321086:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d1bc4700 x1876350913387776/t4294973848(4294973848) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431432.469796:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66e:0x0](ffff9811c4823248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.469807:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603, flags=258 00000080:00200000:3.0:1789431432.469816:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 44 00000080:00200008:3.0:1789431432.469825:0:321086:0:(file.c:1090:ll_file_open()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), flags 704400 00000002:00010000:3.0:1789431432.469900:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66e:0x0], intent: open flags 0104240704401 00010000:00010000:3.0:1789431432.469970:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.469980:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431432.469990:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431432.469997:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913388416/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431432.470006:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.470025:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913388416:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.470192:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8d58e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.473089:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913388416:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.473165:0:321086:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.473174:0:321086:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.473221:0:321086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.473229:0:321086:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.473244:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9811d1bc4a80 x1876350913388416/t4294973850(4294973850) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431432.473261:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4a80 x1876350913388416/t4294973850(4294973850) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431432.473326:0:321086:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.473338:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 16 res 8589935618/1647 (0 0) 00000080:00200000:3.0:1789431432.473344:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x66e:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.473366:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x66f:0x0], updating i_size 4096 00000080:00010000:3.0:1789431432.473371:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) for lock 0x1ffb2ee1bd27dca7 00000080:00010000:3.0:1789431432.473380:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dca7 from it ffffbc42c2dbbbd0 00010000:00010000:3.0:1789431432.473384:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.473391:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1789431432.473413:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4a80 x1876350913388416/t4294973850(4294973850) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1789431432.473464:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603:[0x200000402:0x66f:0x0](ffff9811c4825348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431432.473470:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348),name=8cb3b20c9c16f4cfd9b0aa58b7b86072f6abb6e8b2a2ad8ed4e17e4e0d07858bbb0a4739c929a764311e0615a241fc91abf662a5f7f720babe4e7620d8a6603 00000002:00010000:3.0:1789431432.473486:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x66f:0x0]) in obj [0x200000402:0x66e:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.473494:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431432.473502:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431432.473506:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x66e:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431432.473509:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) 00000080:00010000:3.0:1789431432.473512:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x66f:0x0](ffff9811c4825348) for lock 0x1ffb2ee1bd27dc99 00000080:00010000:3.0:1789431432.473514:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dc99 from it ffffbc42c2dbbcd8 00010000:00010000:3.0:1789431432.473516:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431432.473520:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27dc99 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa738 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1789431432.473546:0:321086:0:(file.c:479:ll_file_release()) START file 8a486aa249fb863429f04e473b623e746e54aa5a48815a7b99fde9d3fcd62c20723aad7a2aabf90af2950f2a2585d74f7ce2ae6e7af2e563288a7132b0997ea:[0x200000402:0x66a:0x0](ffff9811c4835b88), flags 704000 00010000:00010000:3.0:1789431432.473660:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40000/0x1ffb2ee1bd27dc3e lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x66a:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa65f expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431432.473679:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1789431432.473699:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:3.0:1789431432.473712:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66f:0x0](ffff9811c4825348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.473718:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=9ccc22f15315299866b57621dbae0af11161de1f7c6bbc6fb46bb7d8b2ccfd5bfc37c6d45d89865761c1f9ab16a21efbce622aa8185bc5ff900a1ae78ca4d48, flags=0 00000080:00200008:3.0:1789431432.473724:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 9ccc22f15315299866b57621dbae0af11161de1f7c6bbc6fb46bb7d8b2ccfd5bfc37c6d45d89865761c1f9ab16a21efbce622aa8185bc5ff900a1ae78ca4d48:[0x200000402:0x670:0x0](ffff9811c4833248), request_mask 2047, flags 0, foreign 0 00000100:00100000:3.0:1789431432.535490:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d3a55c00 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913389568:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.535516:0:321086:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535531:0:321086:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x671:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535539:0:321086:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535549:0:321086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535558:0:321086:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535570:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535582:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535589:0:321086:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.535598:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d3a55c00 x1876350913389568/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431448 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431432.535610:0:321086:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535618:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1649 (0 0) 00000080:00200000:3.0:1789431432.535624:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x670:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.535644:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x671:0x0], updating i_size 4096 00000080:00010000:3.0:1789431432.535649:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) for lock 0x1ffb2ee1bd27dcc3 00000080:00010000:3.0:1789431432.535658:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) 00000080:00010000:3.0:1789431432.535661:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dcc3 from it ffffbc42c2dbbcb8 00010000:00010000:3.0:1789431432.535664:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.535672:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431432.535728:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x670:0x0](ffff9811c4833248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.535736:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc, flags=258 00000080:00200000:3.0:1789431432.535755:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348), inode mode 41ed mask 44 00000080:00200008:3.0:1789431432.535763:0:321086:0:(file.c:1090:ll_file_open()) START file 6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc:[0x200000402:0x671:0x0](ffff9811c4835348), flags 704400 00000002:00010000:3.0:1789431432.535785:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x671:0x0]) in obj [0x200000402:0x670:0x0], intent: open flags 0100240704401 00010000:00010000:3.0:1789431432.535826:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.536552:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431432.536564:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431432.536573:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4700 x1876350913389824/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431432.536581:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.536601:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913389824:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.536764:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.552217:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913389824:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.552241:0:321086:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552266:0:321086:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552274:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552281:0:321086:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552290:0:321086:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552296:0:321086:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431432.552300:0:321086:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552307:0:321086:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x206400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552315:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 5/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x20e400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552322:0:321086:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431432.552327:0:321086:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.552333:0:321086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcca lrc: 0/0,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x1 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.552351:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff9811d1bc4700 x1876350913389824/t4294973854(4294973854) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431432.552364:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4700 x1876350913389824/t4294973854(4294973854) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431432.552377:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x670:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.552396:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x671:0x0], updating i_size 4096 00000002:00100000:3.0:1789431432.552414:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4700 x1876350913389824/t4294973854(4294973854) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1789431432.552479:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc:[0x200000402:0x671:0x0](ffff9811c4835348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431432.552486:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348),name=6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc 00000002:00010000:3.0:1789431432.552505:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x671:0x0]) in obj [0x200000402:0x670:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.552520:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431432.552530:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431432.552537:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x670:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431432.552542:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) 00000080:00010000:3.0:1789431432.552545:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) for lock 0x1ffb2ee1bd27dcc3 00000080:00010000:3.0:1789431432.552550:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dcc3 from it ffffbc42c2dbbcd8 00010000:00010000:3.0:1789431432.552553:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431432.552561:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431432.552587:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348) pos/size0/4096 32bit_api 0 00000002:00010000:3.0:1789431432.552598:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x671:0x0]) in obj [0x200000402:0x671:0x0], intent: readdir flags 0 00010000:00010000:3.0:1789431432.552603:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431432.552610:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1789431432.553228:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913389952:192.168.203.128@tcp:37:rm.0 00000100:00100000:3.0:1789431432.553316:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.567447:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913389952:192.168.203.128@tcp:37:rm.0 00010000:00010000:3.0:1789431432.568056:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431432.568071:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:3.0:1789431432.568134:0:321086:0:(file.c:479:ll_file_release()) START file 6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc:[0x200000402:0x671:0x0](ffff9811c4835348), flags 704000 00000002:00100000:3.0:1789431432.568193:0:321086:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d1bc4700 x1876350913389824/t4294973854(4294973854) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431432.568205:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431432.568211:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5500 x1876350913390208/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431432.568231:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913390208:192.168.203.128@tcp:35:rm.0 00000100:00100000:3.0:1789431432.568308:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.571496:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913390208:192.168.203.128@tcp:35:rm.0 00000002:00100000:3.0:1789431432.571528:0:321086:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d1bc4700 x1876350913389824/t4294973854(4294973854) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431432.571589:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x670:0x0](ffff9811c4833248), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.571599:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc, flags=258 00000080:00200000:3.0:1789431432.571609:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348), inode mode 41ed mask 44 00000080:00200008:3.0:1789431432.571618:0:321086:0:(file.c:1090:ll_file_open()) START file 6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc:[0x200000402:0x671:0x0](ffff9811c4835348), flags 704400 00000002:00010000:3.0:1789431432.571637:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x671:0x0]) in obj [0x200000402:0x670:0x0], intent: open flags 0104240704401 00010000:00010000:3.0:1789431432.571693:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.571703:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431432.571714:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431432.571721:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5500 x1876350913390464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431432.571730:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x670:0x0].0x0 bits 0x1/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.571749:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913390464:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.571980:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.578608:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913390464:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.578640:0:321086:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x671:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.578650:0:321086:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.578661:0:321086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.578676:0:321086:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.578692:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff9811d1bc5500 x1876350913390464/t4294973856(4294973856) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431432.578706:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc5500 x1876350913390464/t4294973856(4294973856) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431432.578716:0:321086:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.578726:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 16 res 8589935618/1649 (0 0) 00000080:00200000:3.0:1789431432.578731:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x670:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.578750:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x671:0x0], updating i_size 4096 00000080:00010000:3.0:1789431432.578755:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) for lock 0x1ffb2ee1bd27dcd1 00000080:00010000:3.0:1789431432.578764:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dcd1 from it ffffbc42c2dbbbd0 00010000:00010000:3.0:1789431432.578768:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.578777:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27dcd1 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x671:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa7b6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1789431432.578942:0:321086:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc5500 x1876350913390464/t4294973856(4294973856) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1789431432.579005:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc:[0x200000402:0x671:0x0](ffff9811c4835348), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431432.579015:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348),name=6214a5c4a5bcafde433f47d5fff37859275fd9591727313aea6ca54b38c3303ad2028ac7c80f418a6f8d834bab225c58a74a0658105a7c2f419addc5d0e0acc 00000002:00010000:3.0:1789431432.579035:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x671:0x0]) in obj [0x200000402:0x670:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.579046:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431432.579057:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431432.579064:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x670:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431432.579070:0:321086:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) 00000080:00010000:3.0:1789431432.579073:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x671:0x0](ffff9811c4835348) for lock 0x1ffb2ee1bd27dcc3 00000080:00010000:3.0:1789431432.579077:0:321086:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27dcc3 from it ffffbc42c2dbbcd8 00010000:00010000:3.0:1789431432.579081:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431432.579088:0:321086:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40800/0x1ffb2ee1bd27dcc3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x671:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa79a expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1789431432.579121:0:321086:0:(file.c:479:ll_file_release()) START file e57e584b96a4aed132073d894e82def875f555cbd63166f47cd3ce461acd5c96e18fa76c68aeed5175ec07adf896f1688cd197c9dc2504c617efc0960b163cb:[0x200000402:0x66c:0x0](ffff9811c48321c8), flags 704000 00010000:00010000:3.0:1789431432.579140:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0b40600/0x1ffb2ee1bd27dc68 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x66c:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa6c1 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431432.579160:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1789431432.579187:0:321086:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:3.0:1789431432.579205:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x671:0x0](ffff9811c4835348), inode mode 41ed mask 1 00000080:00200000:3.0:1789431432.579213:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=325e89ae1ddd43b3c4a79d9ae3139c81dc4e6b33e96d3f433fb45962b2b86838779d5df8e3f34003e3ddba4776607d38139cdfbdecae714aec87cb862a38933, flags=0 00000080:00200008:3.0:1789431432.579221:0:321086:0:(file.c:6296:ll_getattr_dentry()) START file 325e89ae1ddd43b3c4a79d9ae3139c81dc4e6b33e96d3f433fb45962b2b86838779d5df8e3f34003e3ddba4776607d38139cdfbdecae714aec87cb862a38933:[0x200000402:0x672:0x0](ffff9811c4833a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431432.771807:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x676:0x0](ffff9811c48342c8),name=f999b268f496b67bef31140050f50b268d83fa2409e152b1c977a2f64d4b6b6c2a8127f9790dd42561685eb3c58cfdd2e88364c4675e3541ef8427029f07182 00000002:00010000:2.0:1789431432.771858:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x676:0x0]) in obj [0x200000402:0x675:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431432.771869:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1654 (0 0) 00010000:00010000:2.0:1789431432.771929:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27dd2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x675:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.771939:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27dd2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x675:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.771948:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27dd2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x675:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431432.771971:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913394688:192.168.203.128@tcp:101:rm.0 00000100:00100000:2.0:1789431432.772048:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64600 going to sleep for 11 seconds 00000080:00200000:2.0:1789431432.842598:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x678:0x0](ffff9811c4807448),name=72f5afd309429b4e2b5b1c0c01d07b8a9c0e6d164a564b7f07be7cdae926672d91bedcb5a46d6ca358421d6d776f1477c4beb4275a09ff37384f1767736e626 00000002:00010000:2.0:1789431432.842613:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x678:0x0]) in obj [0x200000402:0x677:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431432.842620:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1656 (0 0) 00010000:00010000:2.0:1789431432.842658:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27dd56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.842666:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27dd56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.842674:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27dd56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x677:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431432.842693:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913396736:192.168.203.128@tcp:101:rm.0 00000100:00100000:2.0:1789431432.842756:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a640c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431432.886423:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x679:0x0](ffff9811c37a4b08),name=211430ca1a13832fa494baa06a12d1fc7a8df7325ac3c978207bedeb6f3da23405dcbad3e32decc61fd3d0999a9dfe524999ac7f2b841c17e32d6d0302d4ac9 00000002:00010000:2.0:1789431432.886443:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x679:0x0]) in obj [0x200000402:0x678:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431432.886453:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1657 (0 0) 00010000:00010000:2.0:1789431432.886495:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27dd6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x678:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.886503:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27dd6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x678:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431432.886512:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27dd6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x678:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431432.886532:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913397760:192.168.203.128@tcp:101:rm.0 00000100:00100000:2.0:1789431432.887687:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64300 going to sleep for 11 seconds 00000080:00200000:3.0:1789431432.924133:0:321086:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x67a:0x0](ffff9811c4875348),name=04a0e70005c09b943f2dbc3e56e8d1b573301efc3114d5c8926b173bb4d4d149e36ed85a904ef8b12fa0386f9f59a5a2ce39751667849aed4a7d8ac724456d6 00000002:00010000:3.0:1789431432.924148:0:321086:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x67a:0x0]) in obj [0x200000402:0x679:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431432.924157:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1658 (0 0) 00010000:00010000:3.0:1789431432.924193:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x679:0x0].0x0 bits 0x0/0x0 rrc: 5 type: IBT flags: 0x10000000000000 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.924207:0:321086:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x679:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.924216:0:321086:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x679:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT flags: 0x0 pid: 321086 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431432.924235:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913398784:192.168.203.128@tcp:101:rm.0 00000100:00100000:3.0:1789431432.924308:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c50fe540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431432.929862:0:321086:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913398784:192.168.203.128@tcp:101:rm.0 00010000:00010000:3.0:1789431432.929891:0:321086:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x679:0x0].0x0 bits 0x2/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929919:0:321086:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929928:0:321086:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929944:0:321086:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929955:0:321086:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929968:0:321086:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929976:0:321086:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.929982:0:321086:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431432.929994:0:321086:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4700 x1876350913398784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431448 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431432.930015:0:321086:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27dd80 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x67a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa953 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431432.930024:0:321086:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1658 (0 0) 00000080:00200000:3.0:1789431432.930030:0:321086:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x679:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431432.930050:0:321086:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x67a:0x0], updating i_size 4096 00000080:00010000:3.0:1789431432.930055:0:321086:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x67a:0x0](ffff9811c4875348) for lock 0x1ffb2ee1bd27dd80 00010000:00010000:2.0:1789431433.491519:0:321086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 1/0,0 mode: --/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431433.491539:0:321086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2e00/0x1ffb2ee1bd27db1f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa2bc expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431433.491546:0:321086:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dc84 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x66e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa707 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431433.491552:0:321086:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:2.0:1789431433.491556:0:321086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dca7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x66f:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa754 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431433.491569:0:321086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2e00/0x1ffb2ee1bd27db1f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x66f:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa2bc expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431433.491584:0:321086:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27dc84 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x66e:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aa707 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000020:00100000:2.0:1789431433.491601:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431433.491614:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913417600/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431433.491639:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913417600:192.168.203.128@tcp:36:rm.0 00000100:00100000:2.0:1789431433.491736:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a640c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431433.500129:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, flags=0 00000080:00200000:2.0:1789431433.500138:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x66d:0x0](ffff9811c4834b08), inode mode 41ed mask 3 00000080:00200000:2.0:1789431433.500143:0:321086:0:(namei.c:2254:ll_rmdir()) VFS Op:name=6a79618e8b0a8d371415ac72cc9f4053fc98a51b2fcba029d51a3c58dd7ba211dcfc6a2c1cd33ea2dcfe3a5d2dbb2cc6b0466b4037e166e9906f7b19e00a5fa, dir=[0x200000402:0x66d:0x0](ffff9811c4834b08) 00010000:00010000:2.0:1789431433.500162:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d009ee00/0x1ffb2ee1bd27dc6f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66d:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aa6d6 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431433.500177:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c8c9e000/0x1ffb2ee1bd27db11 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x66d:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa284 expref: -99 pid: 321067 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431433.500193:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d009e200/0x1ffb2ee1bd27dc92 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x66e:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aa723 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1789431433.500233:0:321086:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d0bdce00 x1876350913387392/t4294973847(4294973847) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431448 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1789431433.500254:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1789431433.500259:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101c00 x1876350913417856/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1789431433.500279:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913417856:192.168.203.128@tcp:35:rm.0 00000100:00100000:2.0:1789431433.500349:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a646c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431433.632452:0:321086:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x665:0x0](ffff9811c4870908) 00000080:00200008:2.0:1789431433.632509:0:321086:0:(file.c:479:ll_file_release()) START file 2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8:[0x200000402:0x664:0x0](ffff9811c4872a08), flags 704000 00010000:00010000:2.0:1789431433.632534:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3c00/0x1ffb2ee1bd27dbc0 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x664:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aa539 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431433.632564:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431433.632572:0:321086:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8, flags=0 00000080:00200000:2.0:1789431433.632580:0:321086:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431433.632584:0:321086:0:(namei.c:2254:ll_rmdir()) VFS Op:name=2316eb1167be62737171248769a250e2f5f7a709ff13fca0c8f3601159ba164e8f6ffaa667ffd96d79796b7bfbd4208b4391c6d96b21eebd53738fcdcc63ce8, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:2.0:1789431433.632607:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27daa1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aa0ee expref: -99 pid: 321065 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431433.632621:0:321086:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431433.632660:0:321086:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3c00/0x1ffb2ee1bd27dbc0 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x664:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aa539 expref: -99 pid: 321086 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:2.0:1789431433.632697:0:321086:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811c8d45880 x1876350913377152/t4294973817(4294973817) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 648/608 e 0 to 0 dl 1789431447 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1789431433.632708:0:321086:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1789431433.632713:0:321086:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913422976/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1789431433.632735:0:321086:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job rm:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:321086:1876350913422976:192.168.203.128@tcp:35:rm.0 00000100:00100000:2.0:1789431433.632910:0:321086:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64cc0 going to sleep for 11 seconds 00010000:00010000:2.0:1789431433.895462:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431434.923757:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431435.211038:0:321278:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:3.0:1789431435.965239:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431436.968444:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431437.990289:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431439.015182:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431440.042027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431441.061987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431442.089599:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431443.114316:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431443.654406:0:321994:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_04: symlink returns ENOENT if a component of the name2 path prefix does not exist ========================================================== 20:17:22 (1789431442) 00010000:00010000:2.0:1789431444.144441:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431444.262280:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913424000:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431444.262365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913424256:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431444.262397:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789431444.262403:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913424128:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431444.270236:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913424128:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431444.270256:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913424000:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431444.270269:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2103800 x1876350913424256/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431460 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431444.270278:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913424256:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431445.159661:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431446.182087:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431447.208016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431448.698333:0:322309:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/04.t against lustre filesystem 00000080:00200000:2.0:1789431448.761993:0:322312:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431448.762018:0:322312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431448.762035:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431448.762051:0:322312:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431448.762064:0:322312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431448.762071:0:322312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431448.762075:0:322312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27d8c5 00000080:00010000:2.0:1789431448.762081:0:322312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d8c5 from it ffffbc42c0d9bb58 00010000:00010000:2.0:1789431448.762085:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431448.762093:0:322312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1789431448.762107:0:322312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431448.762119:0:322312:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=13 00000080:00200000:2.0:1789431448.762123:0:322312:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:2.0:1789431448.762135:0:322312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431448.762177:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 322312 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431448.762185:0:322312:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 322312 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431448.762194:0:322312:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 322312 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431448.762218:0:322312:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322312:1876350913424384:192.168.203.128@tcp:101:lfs.0 00000100:00100000:2.0:1789431448.762550:0:322312:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0bcd780 going to sleep for 11 seconds 00000100:00100000:3.0:1789431448.791123:0:322312:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322312:1876350913424384:192.168.203.128@tcp:101:lfs.0 00010000:00010000:3.0:1789431448.791148:0:322312:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791164:0:322312:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791173:0:322312:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791185:0:322312:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791196:0:322312:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791208:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791217:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791225:0:322312:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431448.791237:0:322312:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2101c00 x1876350913424384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431448.791251:0:322312:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791260:0:322312:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431448.791266:0:322312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431448.791290:0:322312:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1789431448.791293:0:322312:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:3.0:1789431448.791299:0:322312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431448.791309:0:322312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431448.791312:0:322312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c0d9bb20 00010000:00010000:3.0:1789431448.791315:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.791324:0:322312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1789431448.791360:0:322312:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1789431448.791366:0:322312:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431448.791386:0:322312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431448.791395:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431448.791405:0:322312:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431448.791412:0:322312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431448.791417:0:322312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431448.791420:0:322312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431448.791424:0:322312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c0d9bcb8 00010000:00010000:3.0:1789431448.791427:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431448.791436:0:322312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:3.0:1789431448.792013:0:322312:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1789431448.792019:0:322312:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431448.792031:0:322312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431448.792042:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1789431448.792053:0:322312:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1789431448.792061:0:322312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431448.792066:0:322312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431448.792069:0:322312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27d8c5 00000080:00010000:3.0:1789431448.792073:0:322312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d8c5 from it ffffbc42c0d9bcb8 00010000:00010000:3.0:1789431448.792077:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1789431448.792084:0:322312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1789431448.792188:0:322312:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431448.792196:0:322312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431448.792203:0:322312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1789431448.792210:0:322312:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1789431448.792216:0:322312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431448.792220:0:322312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431448.792222:0:322312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27d8c5 00000080:00010000:3.0:1789431448.792226:0:322312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27d8c5 from it ffffbc42c0d9bbb0 00010000:00010000:3.0:1789431448.792229:0:322312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1789431448.792236:0:322312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1789431448.792245:0:322312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:3.0:1789431448.792252:0:322312:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:3.0:1789431448.792268:0:322312:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7fff02ea8200 00000080:00200000:3.0:1789431448.792280:0:322312:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:3.0:1789431448.792300:0:322312:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431448.792340:0:322312:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431448.792347:0:322312:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431448.792350:0:322312:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27d8c5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795a97e8 expref: -99 pid: 318300 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00100000:3.0:1789431448.792370:0:322312:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431448.792379:0:322312:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7480 x1876350913424640/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431448.792405:0:322312:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322312:1876350913424640:192.168.203.128@tcp:36:lfs.0 00000100:00100000:3.0:1789431448.792494:0:322312:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000100:00100000:3.0:1789431448.800460:0:322312:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322312:1876350913424640:192.168.203.128@tcp:36:lfs.0 00000080:00200008:3.0:1789431448.800555:0:322312:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000100:00100000:3.0:1789431448.950037:0:322322:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913425024:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431448.950060:0:322322:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950077:0:322322:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950084:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950091:0:322322:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950099:0:322322:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950105:0:322322:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c0030200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431448.950109:0:322322:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950115:0:322322:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950121:0:322322:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950127:0:322322:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431448.950242:0:322322:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950251:0:322322:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27de4b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431448.950267:0:322322:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd4700 x1876350913425024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431464 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431448.950286:0:322322:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431448.950535:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431448.950549:0:322322:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431448.950557:0:322322:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1beb970 00010000:00010000:3.0:1789431448.950561:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431448.950569:0:322322:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431448.950621:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.950629:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431448.950641:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.950644:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431448.950653:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.950656:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431448.950664:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.950673:0:322322:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431448.950677:0:322322:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431448.950692:0:322322:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431448.950728:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950736:0:322322:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.950746:0:322322:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431448.950765:0:322322:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913425152:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431448.952053:0:322322:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000100:00100000:3.0:1789431448.968973:0:322322:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913425152:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431448.968994:0:322322:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969006:0:322322:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969013:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969019:0:322322:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969026:0:322322:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969031:0:322322:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431448.969035:0:322322:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969041:0:322322:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969048:0:322322:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969055:0:322322:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431448.969059:0:322322:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969065:0:322322:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27de52 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431448.969079:0:322322:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7480 x1876350913425152/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431464 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431448.969092:0:322322:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431448.969117:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431448.969129:0:322322:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431448.969137:0:322322:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1beb970 00010000:00010000:3.0:1789431448.969141:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431448.969149:0:322322:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431448.969204:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.969211:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431448.969223:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.969226:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431448.969233:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431448.969244:0:322322:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf0f00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431448.969250:0:322322:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431448.969266:0:322322:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431448.969307:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27de59 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431448.969314:0:322322:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27de59 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431448.969323:0:322322:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431448.969329:0:322322:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7480 x1876350913425408/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431448.969337:0:322322:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27de59 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431448.969354:0:322322:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913425408:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431448.969584:0:322322:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00010000:00010000:2.0:1789431449.254040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789431449.257721:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.257736:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431449.257750:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.257754:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.257764:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.257990:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.258005:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258009:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.258018:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258022:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.258030:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258034:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.258043:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258046:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.258053:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258056:0:322322:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.258064:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.258076:0:322322:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf0c00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431449.258083:0:322322:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431449.258107:0:322322:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: librt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431449.258171:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.258182:0:322322:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431449.258194:0:322322:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431449.258206:0:322322:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5180 x1876350913427456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431449.258217:0:322322:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431449.258237:0:322322:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913427456:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431449.258461:0:322322:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000100:00100000:3.0:1789431449.270827:0:322322:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913427456:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431449.270878:0:322322:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270892:0:322322:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270904:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270911:0:322322:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270919:0:322322:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270925:0:322322:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431449.270930:0:322322:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270938:0:322322:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270945:0:322322:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270951:0:322322:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431449.270958:0:322322:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.270964:0:322322:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de91 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431449.270984:0:322322:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc5180 x1876350913427456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431465 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431449.270998:0:322322:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431449.271023:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431449.271036:0:322322:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00010000:3.0:1789431449.271045:0:322322:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1beb9a0 00010000:00010000:3.0:1789431449.271049:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431449.271057:0:322322:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1789431449.335642:0:322322:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.335666:0:322322:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:3.0:1789431449.335671:0:322322:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:3.0:1789431449.335698:0:322322:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .proverc,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431449.335755:0:322322:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.335766:0:322322:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.335776:0:322322:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431449.335813:0:322322:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913427584:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431449.336342:0:322322:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000100:00100000:3.0:1789431449.350046:0:322322:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322322:1876350913427584:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431449.350069:0:322322:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350092:0:322322:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350100:0:322322:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350107:0:322322:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350116:0:322322:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350122:0:322322:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431449.350126:0:322322:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350132:0:322322:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350140:0:322322:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350146:0:322322:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431449.350152:0:322322:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.350159:0:322322:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27de98 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 322322 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431449.350175:0:322322:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d1bc5180 x1876350913427584/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431465 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431449.350251:0:322322:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000100:00100000:2.0:1789431449.383260:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913427840:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431449.383343:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789431449.383350:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913427968:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431449.394130:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2101880 x1876350913427968/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431465 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431449.394149:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913427968:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431449.394162:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913427840:192.168.203.128@tcp:400:kworker.0 00000080:00200000:2.0:1789431449.869206:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431449.869219:0:322327:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431449.869225:0:322327:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431449.869242:0:322327:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431449.869256:0:322327:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1789431449.869269:0:322327:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1789431449.869276:0:322327:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431449.869283:0:322327:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431449.869286:0:322327:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:2.0:1789431449.869292:0:322327:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431449.869296:0:322327:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1789431449.869304:0:322327:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1789431449.870686:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870698:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431449.870710:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870714:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.870723:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870726:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.870735:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870738:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.870747:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870750:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.870759:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870762:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.870770:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870773:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.870794:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870798:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.870810:0:322327:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.870814:0:322327:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431449.878244:0:322330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431449.878259:0:322330:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431449.878266:0:322330:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431449.878292:0:322330:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431449.878309:0:322330:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1789431449.878324:0:322330:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1789431449.878333:0:322330:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431449.878339:0:322330:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431449.878342:0:322330:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431449.878349:0:322330:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1563cb8 00010000:00010000:3.0:1789431449.878353:0:322330:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1789431449.878362:0:322330:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1789431449.884476:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431449.884494:0:322333:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431449.884502:0:322333:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431449.884526:0:322333:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431449.884545:0:322333:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1789431449.884560:0:322333:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:2.0:1789431449.884568:0:322333:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431449.884575:0:322333:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431449.884579:0:322333:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:2.0:1789431449.884595:0:322333:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431449.884599:0:322333:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:2.0:1789431449.884608:0:322333:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:3.0:1789431449.885613:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431449.885622:0:322335:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431449.885627:0:322335:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431449.885640:0:322335:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431449.885650:0:322335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431449.885661:0:322335:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:3.0:1789431449.885669:0:322335:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431449.885674:0:322335:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431449.885677:0:322335:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431449.885682:0:322335:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c160bcb8 00010000:00010000:3.0:1789431449.885685:0:322335:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431449.885694:0:322335:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1789431449.887395:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887402:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431449.887416:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887420:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.887432:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887436:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.887447:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887451:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.887460:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887464:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.887475:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887478:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.887486:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887489:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.887497:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887500:0:322335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.887510:0:322335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.887519:0:322335:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf0000, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431449.887527:0:322335:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431449.887548:0:322335:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libsigsegv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431449.887609:0:322335:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27debb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322335 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.887620:0:322335:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27debb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322335 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431449.887631:0:322335:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431449.887643:0:322335:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5f80 x1876350913428736/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431449.887653:0:322335:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27debb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322335 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431449.887673:0:322335:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322335:1876350913428736:192.168.203.128@tcp:101:awk.0 00000080:00200000:2.0:1789431449.888039:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888046:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431449.888056:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888059:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.888069:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888072:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.888080:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888084:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.888093:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888096:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.888104:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888107:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.888114:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888117:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.888131:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888134:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.888144:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.888148:0:322333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:3.0:1789431449.888601:0:322335:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000080:00200000:2.0:1789431449.889568:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431449.889579:0:322333:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431449.889585:0:322333:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431449.890754:0:322333:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431449.890765:0:322333:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1789431449.890779:0:322333:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1789431449.890796:0:322333:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431449.890804:0:322333:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431449.890809:0:322333:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:2.0:1789431449.890814:0:322333:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431449.890817:0:322333:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:2.0:1789431449.890842:0:322333:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1789431449.890866:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.890870:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:2.0:1789431449.890876:0:322333:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:2.0:1789431449.890898:0:322333:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:2.0:1789431449.890914:0:322333:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:2.0:1789431449.891695:0:322333:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431449.891701:0:322333:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431449.891710:0:322333:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431449.891719:0:322333:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431449.891729:0:322333:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431449.891736:0:322333:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431449.891741:0:322333:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431449.891744:0:322333:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27de3d 00000080:00010000:2.0:1789431449.891748:0:322333:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de3d from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431449.891752:0:322333:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431449.891759:0:322333:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431449.891778:0:322333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.891794:0:322333:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:2.0:1789431449.891799:0:322333:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3797, max_age 3823 00800000:00000004:2.0:1789431449.891807:0:322333:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3797, max_age 3823 00000100:00100000:2.0:1789431449.891828:0:322333:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322333:1876350913429120:192.168.203.128@tcp:41:df.0 00000100:00100000:2.0:1789431449.891884:0:322333:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0bcd900 going to sleep for 11 seconds 00000100:00100000:2.0:1789431449.897734:0:322333:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322333:1876350913429120:192.168.203.128@tcp:41:df.0 00800000:00000004:2.0:1789431449.897750:0:322333:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:2.0:1789431449.897758:0:322333:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:2.0:1789431449.897763:0:322333:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000080:00200000:2.0:1789431449.936864:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431449.936887:0:322338:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431449.936894:0:322338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431449.936919:0:322338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431449.936937:0:322338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1789431449.936951:0:322338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1789431449.936960:0:322338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431449.936967:0:322338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431449.936970:0:322338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:2.0:1789431449.936978:0:322338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c163fcb8 00010000:00010000:2.0:1789431449.936981:0:322338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1789431449.937059:0:322338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:3.0:1789431449.937211:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431449.937218:0:322339:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431449.937223:0:322339:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431449.937234:0:322339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431449.937243:0:322339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:3.0:1789431449.937252:0:322339:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:3.0:1789431449.937259:0:322339:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431449.937263:0:322339:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431449.937266:0:322339:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431449.937269:0:322339:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c1677cb8 00010000:00010000:3.0:1789431449.937273:0:322339:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:3.0:1789431449.937280:0:322339:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1789431449.938915:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938922:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431449.938931:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938935:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.938942:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938945:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.938953:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938956:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.938965:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938968:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431449.938976:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.938980:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.939051:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.939057:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.939070:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.939074:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431449.939150:0:322338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431449.939155:0:322338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431449.939456:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939462:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431449.939470:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939473:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.939482:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939485:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.939497:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939501:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.939512:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939515:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431449.939528:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939531:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.939541:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939544:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.939554:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939557:0:322339:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431449.939568:0:322339:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431449.939578:0:322339:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0a2aa80, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431449.939585:0:322339:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431449.939600:0:322339:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libssl.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431449.939654:0:322339:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ded7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322339 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431449.939662:0:322339:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ded7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322339 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431449.939673:0:322339:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431449.939684:0:322339:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5c00 x1876350913429888/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431449.939694:0:322339:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ded7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322339 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431449.939715:0:322339:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322339:1876350913429888:192.168.203.128@tcp:101:openssl.0 00000100:00100000:3.0:1789431449.940018:0:322339:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1498900 going to sleep for 11 seconds 00000080:00200000:3.0:1789431450.001394:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431450.001411:0:322344:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431450.001418:0:322344:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431450.001442:0:322344:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431450.001460:0:322344:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:3.0:1789431450.001475:0:322344:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:3.0:1789431450.001483:0:322344:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431450.001490:0:322344:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431450.001493:0:322344:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27de36 00000080:00010000:3.0:1789431450.001500:0:322344:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de36 from it ffffbc42c174bcb8 00010000:00010000:3.0:1789431450.001504:0:322344:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:3.0:1789431450.001513:0:322344:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:3.0:1789431450.004109:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004124:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.004138:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004142:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004150:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004152:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004160:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004163:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004171:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004173:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004183:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004187:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.004197:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004200:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.004210:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004213:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.004223:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004226:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:3.0:1789431450.004415:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004420:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.004428:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004431:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004439:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004442:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004449:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004451:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004456:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004459:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.004465:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004468:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.004474:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004476:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.004482:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.004484:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.005105:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005117:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:3.0:1789431450.005323:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005329:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.005338:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005342:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.005351:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005354:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.005362:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005364:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.005372:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005375:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.005383:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005386:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.005395:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005398:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.005406:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005408:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.005416:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005420:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:3.0:1789431450.005714:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005720:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.005731:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.005734:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.005742:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006038:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.006061:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006065:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.006081:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006084:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.006092:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006096:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.006105:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006108:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.006116:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006119:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.006127:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006131:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431450.006256:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006262:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.006272:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006276:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.006286:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.006289:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007041:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007049:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007067:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007069:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007074:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007077:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007082:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007084:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007088:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007090:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007095:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007097:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431450.007219:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007223:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.007230:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007242:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007250:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007252:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007262:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007264:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007275:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007278:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.007289:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007291:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007303:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007305:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007315:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007318:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.007335:0:322344:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.007338:0:322344:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431450.049575:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049591:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431450.049603:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049607:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.049614:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049617:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.049623:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049626:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.049633:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049635:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.049641:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049644:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.049652:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049656:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.049665:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049668:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.049676:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.049679:0:322347:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431450.050458:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.050476:0:322347:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_1549e12aa8f20960e45f100786a85152, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431450.050481:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431450.050485:0:322347:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431450.050489:0:322347:0:(namei.c:2165:do_mkdir()) VFS Op:name=pjdfstest_1549e12aa8f20960e45f100786a85152, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:2.0:1789431450.050535:0:322347:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1789431450.050559:0:322347:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431450.050685:0:322347:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:2.0:1789431450.050693:0:322347:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431450.050696:0:322347:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27de36 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aad90 expref: -99 pid: 322312 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000020:00100000:2.0:1789431450.050715:0:322347:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431450.050727:0:322347:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913430656/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 592/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431450.050753:0:322347:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322347:1876350913430656:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431450.051175:0:322347:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0bcd900 going to sleep for 11 seconds 00000100:00100000:2.0:1789431450.058092:0:322347:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322347:1876350913430656:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431450.058143:0:322347:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x686:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431450.058149:0:322347:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x686:0x0], updating i_size 4096 00000080:00200000:3.0:1789431450.129638:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431450.129673:0:322353:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431450.129681:0:322353:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431450.129702:0:322353:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431450.129720:0:322353:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431450.129734:0:322353:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431450.129742:0:322353:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431450.129749:0:322353:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431450.129753:0:322353:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27deec 00000080:00010000:3.0:1789431450.129760:0:322353:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27deec from it ffffbc42c160bcb8 00010000:00010000:3.0:1789431450.129764:0:322353:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431450.129773:0:322353:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431450.131160:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131172:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431450.131183:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131187:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.131194:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131197:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.131205:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131208:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.131215:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131217:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431450.131224:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131227:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.131235:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131238:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.131247:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131250:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431450.131257:0:322353:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431450.131261:0:322353:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431450.152239:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431450.152255:0:322355:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431450.152260:0:322355:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431450.152278:0:322355:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431450.152294:0:322355:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431450.152306:0:322355:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431450.152312:0:322355:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431450.152317:0:322355:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431450.152320:0:322355:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27deec 00000080:00010000:3.0:1789431450.152325:0:322355:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27deec from it ffffbc42c1677cb8 00010000:00010000:3.0:1789431450.152327:0:322355:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431450.152333:0:322355:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27deec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aafb9 expref: -99 pid: 322350 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431450.155683:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155699:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431450.155709:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155712:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.155716:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155718:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.155723:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155725:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.155730:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155732:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.155736:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155738:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.155819:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155824:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.155909:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155913:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.155923:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.155926:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431450.156606:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156614:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431450.156625:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156628:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156635:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156637:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156645:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156647:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156654:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156657:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156664:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156667:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156676:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156679:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156688:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156690:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156697:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156701:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431450.156898:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156903:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431450.156911:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156913:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156920:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156922:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156930:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156932:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156938:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156940:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.156946:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156963:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156974:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156977:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156985:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156987:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.156994:0:322355:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.156997:0:322355:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431450.204331:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431450.204349:0:322360:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431450.204357:0:322360:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431450.204378:0:322360:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431450.204397:0:322360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431450.204414:0:322360:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431450.204423:0:322360:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431450.204430:0:322360:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431450.204435:0:322360:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431450.204442:0:322360:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c178bcb8 00010000:00010000:3.0:1789431450.204447:0:322360:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431450.204457:0:322360:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431450.206315:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206329:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431450.206341:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206346:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.206355:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206359:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.206369:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206373:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.206384:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206387:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431450.206397:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206401:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.206411:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206415:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.206424:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206427:0:322360:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431450.206436:0:322360:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431450.206446:0:322360:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5713e00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431450.206453:0:322360:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431450.206475:0:322360:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431450.206538:0:322360:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27df4e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 322360 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431450.206549:0:322360:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27df4e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322360 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431450.206561:0:322360:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431450.206573:0:322360:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100000 x1876350913433472/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431450.206584:0:322360:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27df4e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 322360 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431450.206606:0:322360:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:322360:1876350913433472:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431450.206777:0:322360:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0bcd900 going to sleep for 11 seconds 00010000:00010000:2.0:1789431450.281083:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431453.351202:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431455.403175:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431458.469990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431461.542939:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431464.230074:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913434624:192.168.203.128@tcp:17:kworker.0 00000100:00100000:3.0:1789431464.233242:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431464.233261:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913434752:192.168.203.128@tcp:17:kworker.0 00010000:00080000:3.0:1789431464.236229:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2103480 x1876350913434624/t0(0) o17->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 456/432 e 0 to 0 dl 1789431480 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431464.236253:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913434624:192.168.203.128@tcp:17:kworker.0 00000100:00100000:2.0:1789431464.253853:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913434752:192.168.203.128@tcp:17:kworker.0 00010000:00010000:2.0:1789431464.615272:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431465.425263:0:323575:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/05.t against lustre filesystem 00000080:00200000:3.0:1789431465.653719:0:323587:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431465.653750:0:323587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431465.653769:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431465.653784:0:323587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431465.653792:0:323587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431465.653800:0:323587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431465.653803:0:323587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27de3d 00000080:00010000:3.0:1789431465.653810:0:323587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de3d from it ffffbc42c1eb3b58 00010000:00010000:3.0:1789431465.653814:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431465.653822:0:323587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431465.654120:0:323587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.654137:0:323587:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=13 00000080:00200000:3.0:1789431465.654141:0:323587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:3.0:1789431465.654153:0:323587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431465.654192:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.654200:0:323587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.654208:0:323587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431465.654230:0:323587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323587:1876350913435264:192.168.203.128@tcp:101:lfs.0 00000100:00100000:3.0:1789431465.654347:0:323587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecc00 going to sleep for 11 seconds 00000100:00100000:3.0:1789431465.665693:0:323587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323587:1876350913435264:192.168.203.128@tcp:101:lfs.0 00010000:00010000:3.0:1789431465.665729:0:323587:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665746:0:323587:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665754:0:323587:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665765:0:323587:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665776:0:323587:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665789:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665796:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665808:0:323587:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431465.665819:0:323587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc5c00 x1876350913435264/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431481 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431465.665864:0:323587:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665877:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431465.665889:0:323587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431465.665898:0:323587:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665904:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665911:0:323587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27df5c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ab125 expref: -99 pid: 323587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.665935:0:323587:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:3.0:1789431465.665942:0:323587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431465.665967:0:323587:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1789431465.665970:0:323587:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:3.0:1789431465.665975:0:323587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431465.665987:0:323587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431465.665991:0:323587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1eb3b20 00010000:00010000:3.0:1789431465.665994:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431465.666002:0:323587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200008:3.0:1789431465.666051:0:323587:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1789431465.666058:0:323587:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431465.666077:0:323587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431465.666084:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431465.666093:0:323587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1789431465.666099:0:323587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431465.666103:0:323587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431465.666106:0:323587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431465.666110:0:323587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1eb3cb8 00010000:00010000:3.0:1789431465.666113:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1789431465.666121:0:323587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200008:3.0:1789431465.666941:0:323587:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1789431465.666948:0:323587:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431465.666959:0:323587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431465.666970:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431465.666981:0:323587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431465.666987:0:323587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431465.666992:0:323587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431465.666995:0:323587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27de3d 00000080:00010000:3.0:1789431465.666999:0:323587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de3d from it ffffbc42c1eb3cb8 00010000:00010000:3.0:1789431465.667004:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431465.667012:0:323587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431465.667378:0:323587:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431465.667389:0:323587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431465.667397:0:323587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431465.667413:0:323587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431465.667419:0:323587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431465.667423:0:323587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431465.667425:0:323587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27de3d 00000080:00010000:3.0:1789431465.667429:0:323587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27de3d from it ffffbc42c1eb3bb0 00010000:00010000:3.0:1789431465.667432:0:323587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431465.667439:0:323587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431465.667447:0:323587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:3.0:1789431465.667455:0:323587:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:3.0:1789431465.667471:0:323587:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffca33fe800 00000080:00200000:3.0:1789431465.667483:0:323587:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:3.0:1789431465.667505:0:323587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431465.667536:0:323587:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431465.667541:0:323587:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431465.667543:0:323587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27de3d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aad9e expref: -99 pid: 321862 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:3.0:1789431465.667559:0:323587:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431465.667567:0:323587:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5c00 x1876350913435648/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431465.667581:0:323587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323587:1876350913435648:192.168.203.128@tcp:36:lfs.0 00000100:00100000:3.0:1789431465.667649:0:323587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431465.681907:0:323587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323587:1876350913435648:192.168.203.128@tcp:36:lfs.0 00000080:00200008:3.0:1789431465.682022:0:323587:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200008:2.0:1789431465.828049:0:323077:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431465.828072:0:323077:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431465.828101:0:323077:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431465.828158:0:323077:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 323077 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431465.828167:0:323077:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 323077 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431465.828177:0:323077:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 323077 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431465.828200:0:323077:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323077:1876350913435776:192.168.203.128@tcp:101:bash.0 00000100:00100000:2.0:1789431465.828275:0:323077:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000080:00200000:3.0:1789431465.846888:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.846911:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431465.847071:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847095:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.847112:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847116:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.847130:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847134:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.847142:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847144:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.847152:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847155:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.847163:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847166:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.847172:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847175:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.847182:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.847195:0:323590:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d8c56300, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431465.847201:0:323590:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431465.847227:0:323590:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431465.847290:0:323590:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431465.847301:0:323590:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431465.847311:0:323590:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431465.847323:0:323590:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100e00 x1876350913436032/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431465.847333:0:323590:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431465.847354:0:323590:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323590:1876350913436032:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431465.847525:0:323590:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431465.869185:0:323590:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323590:1876350913436032:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431465.869217:0:323590:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869234:0:323590:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869243:0:323590:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869252:0:323590:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869261:0:323590:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869268:0:323590:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431465.869272:0:323590:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869280:0:323590:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869288:0:323590:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869296:0:323590:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431465.869303:0:323590:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431465.869310:0:323590:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27df6a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431465.869334:0:323590:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100e00 x1876350913436032/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431481 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431465.869351:0:323590:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431465.869388:0:323590:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431465.869404:0:323590:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00010000:3.0:1789431465.869415:0:323590:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c0d539a0 00010000:00010000:3.0:1789431465.869419:0:323590:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431465.869428:0:323590:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431465.870830:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870887:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431465.870903:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870908:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431465.870917:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870920:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431465.870929:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870932:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431465.870940:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870943:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431465.870951:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870954:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431465.870963:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870966:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431465.870974:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870977:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431465.870985:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431465.870990:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431465.878275:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878296:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431465.878311:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878316:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.878325:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878328:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.878335:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878339:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.878347:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878350:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431465.878357:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878361:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.878368:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878371:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.878379:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878382:0:323590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431465.878390:0:323590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431465.878404:0:323590:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d8c56a80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431465.878412:0:323590:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431465.878448:0:323590:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libresolv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431465.878538:0:323590:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27df71 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431465.878549:0:323590:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27df71 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431465.878563:0:323590:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431465.878587:0:323590:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102a00 x1876350913436288/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431465.878598:0:323590:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27df71 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323590 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431465.879103:0:323590:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323590:1876350913436288:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431465.879390:0:323590:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00010000:00010000:2.0:1789431466.668815:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431466.889632:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1789431466.889659:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00010000:3.0:1789431466.889678:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1ee39a0 00010000:00010000:3.0:1789431466.889684:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:3.0:1789431466.889692:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1789431466.890087:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890098:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431466.890110:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890113:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890121:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890123:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890130:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890132:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890139:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890141:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890148:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890151:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890158:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890160:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890167:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890169:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890175:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890179:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431466.890295:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890301:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431466.890310:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890314:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890322:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890325:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890504:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890509:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890521:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890524:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.890532:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890535:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890543:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890547:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890555:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890558:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.890565:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.890569:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431466.893369:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431466.893398:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.893416:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431466.893430:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431466.893438:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.893444:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431466.893448:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431466.893455:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3b58 00010000:00010000:3.0:1789431466.893459:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431466.893467:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431466.893481:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.893488:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431466.893495:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431466.893500:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431466.893510:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.893518:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1789431466.893526:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:3.0:1789431466.893533:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.893536:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431466.893538:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431466.893541:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431466.893544:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:3.0:1789431466.893551:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:3.0:1789431466.893563:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431466.893567:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431466.893571:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431466.893575:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.893581:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:3.0:1789431466.893588:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:3.0:1789431466.893594:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.893596:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431466.893598:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431466.893601:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431466.893604:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:3.0:1789431466.893610:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200008:3.0:1789431466.893626:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431466.893630:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431466.893635:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.893641:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431466.893648:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431466.893659:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.893662:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431466.893875:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431466.893881:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431466.893885:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431466.893894:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431466.893913:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431466.893920:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.893926:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431466.893995:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431466.894003:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.894008:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431466.894011:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431466.894014:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3b58 00010000:00010000:3.0:1789431466.894017:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431466.894025:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431466.894033:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.894038:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431466.894044:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431466.894047:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431466.894055:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.894062:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:3.0:1789431466.894070:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:3.0:1789431466.894076:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.894080:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431466.894082:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431466.894086:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431466.894090:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:3.0:1789431466.894096:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1789431466.914775:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914798:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431466.914811:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914815:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.914822:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914825:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.914865:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914870:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.914882:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914885:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.914894:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914898:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.914907:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914910:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.914918:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914922:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.914930:0:323596:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.914934:0:323596:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431466.927890:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431466.927902:0:323599:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431466.927908:0:323599:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431466.927925:0:323599:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431466.927939:0:323599:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431466.927951:0:323599:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1789431466.927958:0:323599:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431466.927964:0:323599:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431466.927967:0:323599:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431466.927972:0:323599:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c163fcb8 00010000:00010000:3.0:1789431466.927975:0:323599:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431466.927983:0:323599:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:3.0:1789431466.929810:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.929822:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431466.929954:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.929960:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.929981:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.929985:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.930003:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930007:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.930027:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930030:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.930047:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930051:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.930068:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930072:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.930084:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930088:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.930100:0:323599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.930104:0:323599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431466.979139:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431466.979161:0:323604:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431466.979168:0:323604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431466.979194:0:323604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431466.979213:0:323604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1789431466.979227:0:323604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1789431466.979235:0:323604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431466.979242:0:323604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431466.979246:0:323604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:2.0:1789431466.979253:0:323604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1bfbcb8 00010000:00010000:2.0:1789431466.979257:0:323604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1789431466.979266:0:323604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1789431466.989306:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989316:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431466.989325:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989329:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.989337:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989340:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.989348:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989351:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.989359:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989362:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431466.989368:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989371:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.989377:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989380:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.989388:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989390:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431466.989396:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431466.989404:0:323604:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d8c56300, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431466.989410:0:323604:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431466.989425:0:323604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libssl.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431466.989474:0:323604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431466.989487:0:323604:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431466.989498:0:323604:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431466.989510:0:323604:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102d80 x1876350913439744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431466.989519:0:323604:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431466.989551:0:323604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323604:1876350913439744:192.168.203.128@tcp:101:openssl.0 00000100:00100000:2.0:1789431466.992366:0:323604:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431466.997479:0:323604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323604:1876350913439744:192.168.203.128@tcp:101:openssl.0 00010000:00010000:3.0:1789431466.997501:0:323604:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997515:0:323604:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997521:0:323604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997528:0:323604:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997536:0:323604:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997542:0:323604:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431466.997547:0:323604:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997554:0:323604:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997560:0:323604:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997567:0:323604:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431466.997573:0:323604:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.997580:0:323604:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27dfda lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431466.997598:0:323604:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102d80 x1876350913439744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431482 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431466.997611:0:323604:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431466.997635:0:323604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:3.0:1789431466.997647:0:323604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00010000:3.0:1789431466.997656:0:323604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1bfb9a0 00010000:00010000:3.0:1789431466.997660:0:323604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:3.0:1789431466.997678:0:323604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:3.0:1789431466.998233:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998239:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431466.998254:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998257:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.998269:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998272:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.998281:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998284:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.998295:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998298:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431466.998307:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998310:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.998318:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998321:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.998330:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998333:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431466.998346:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431466.998356:0:323604:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8d06f00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431466.998363:0:323604:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431466.998382:0:323604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypto.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431466.998430:0:323604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dfe1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431466.998443:0:323604:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dfe1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431466.998455:0:323604:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431466.998467:0:323604:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5c00 x1876350913440000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431466.998478:0:323604:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27dfe1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323604 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431466.998504:0:323604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323604:1876350913440000:192.168.203.128@tcp:101:openssl.0 00000100:00100000:3.0:1789431466.998705:0:323604:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec0c0 going to sleep for 11 seconds 00010000:00010000:3.0:1789431467.013074:0:323604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431467.013095:0:323604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00010000:3.0:1789431467.013103:0:323604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1bfb9a0 00010000:00010000:3.0:1789431467.013107:0:323604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431467.013115:0:323604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:3.0:1789431467.013387:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013396:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.013408:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013412:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013421:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013424:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013432:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013435:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013444:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013447:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013454:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013457:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013466:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013469:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013478:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013481:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013489:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013493:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431467.013618:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013623:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.013646:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013650:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013657:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013660:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013679:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013682:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013690:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013693:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.013701:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013705:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013714:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013717:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013725:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013728:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.013735:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.013738:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431467.014158:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014165:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.014175:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014179:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.014186:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014189:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.014197:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014200:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.014208:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014211:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.014218:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014221:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.014229:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014232:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.014241:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014244:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.014252:0:323604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.014255:0:323604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.026776:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.026796:0:323608:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.026803:0:323608:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431467.026829:0:323608:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.026969:0:323608:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:3.0:1789431467.026985:0:323608:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:3.0:1789431467.026994:0:323608:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.027000:0:323608:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431467.027004:0:323608:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431467.027011:0:323608:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1577cb8 00010000:00010000:3.0:1789431467.027014:0:323608:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:3.0:1789431467.027025:0:323608:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1789431467.027720:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.027729:0:323609:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.027734:0:323609:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431467.027745:0:323609:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.027754:0:323609:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1789431467.027764:0:323609:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1789431467.027771:0:323609:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.027775:0:323609:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431467.027778:0:323609:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:2.0:1789431467.027782:0:323609:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c0d9bcb8 00010000:00010000:2.0:1789431467.027796:0:323609:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1789431467.027806:0:323609:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:3.0:1789431467.028721:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028731:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.028741:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028744:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.028752:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028755:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.028762:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028765:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.028772:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028775:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.028782:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028786:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.028793:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028795:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.028802:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028805:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.028812:0:323608:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.028816:0:323608:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.028866:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028871:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.028882:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028885:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.028893:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028897:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.028904:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028907:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.028915:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028917:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.028925:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028929:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.028936:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028938:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.028946:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028950:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.028957:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.028961:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:2.0:1789431467.029114:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029119:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.029128:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029131:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029137:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029139:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029145:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029148:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029154:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029157:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029163:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029166:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029173:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029176:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029183:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029186:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029193:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029197:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431467.029319:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029323:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.029334:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029337:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029347:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029350:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029358:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029360:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029368:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029371:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029378:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029381:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029388:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029391:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029398:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029400:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029408:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029411:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431467.029522:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029527:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.029536:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029539:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029547:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029550:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029557:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029559:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029566:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029569:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029578:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029581:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029590:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029592:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029602:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029605:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029613:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029617:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431467.029736:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029741:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.029749:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029752:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029759:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029761:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029769:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029772:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029782:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029784:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029792:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029795:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029804:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029807:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029814:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029817:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.029824:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029828:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431467.029974:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029980:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.029989:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.029991:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.029998:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030001:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.030007:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030010:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.030017:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030020:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.030026:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030029:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.030036:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030039:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.030046:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030049:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.030058:0:323609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.030061:0:323609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.037125:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037141:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037171:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037175:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037185:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037188:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037201:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037205:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431467.037416:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037422:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.037434:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037437:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037446:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037448:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037458:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037461:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037470:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037473:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037485:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037488:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037498:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037501:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037511:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037514:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037524:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037528:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431467.037660:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037678:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.037690:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037694:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037702:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037705:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037713:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037716:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037725:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037727:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.037735:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037738:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037747:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037750:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037758:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037762:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.037772:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.037776:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431467.038426:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038434:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.038444:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038447:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.038454:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038457:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.038465:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038468:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.038478:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038481:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.038489:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038492:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.038501:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038504:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.038512:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038515:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.038525:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.038528:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.039416:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039424:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.039430:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039432:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.039436:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039438:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.039442:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039444:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.039449:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039450:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.039455:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039457:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.039461:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039463:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.039467:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039468:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.039472:0:323610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.039475:0:323610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431467.070158:0:323614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.070168:0:323614:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.070174:0:323614:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431467.070185:0:323614:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.070195:0:323614:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1789431467.070207:0:323614:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:2.0:1789431467.070214:0:323614:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.070219:0:323614:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431467.070222:0:323614:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:2.0:1789431467.070227:0:323614:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1bfbcb8 00010000:00010000:2.0:1789431467.070230:0:323614:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1789431467.070238:0:323614:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:2.0:1789431467.073792:0:323614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.073803:0:323614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.073819:0:323614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.073824:0:323614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.073864:0:323614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.074264:0:323614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.100197:0:323618:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.100209:0:323618:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.100215:0:323618:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431467.100239:0:323618:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.100258:0:323618:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:3.0:1789431467.100273:0:323618:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:3.0:1789431467.100281:0:323618:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.100287:0:323618:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431467.100291:0:323618:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27df2b 00000080:00010000:3.0:1789431467.100297:0:323618:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df2b from it ffffbc42c1c43cb8 00010000:00010000:3.0:1789431467.100301:0:323618:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:3.0:1789431467.100311:0:323618:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27df2b lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab08b expref: -99 pid: 322358 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:2.0:1789431467.104361:0:323618:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.113648:0:323618:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431467.113658:0:323618:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431467.113692:0:323618:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431467.113751:0:323618:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27dfef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.113761:0:323618:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27dfef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.113771:0:323618:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27dfef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.113794:0:323618:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913440640:192.168.203.128@tcp:101:tail.0 00000100:00100000:2.0:1789431467.114018:0:323618:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.164325:0:323618:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64600 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.171815:0:323618:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441280:192.168.203.128@tcp:101:tail.0 00010000:00010000:3.0:1789431467.171863:0:323618:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171876:0:323618:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171882:0:323618:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171898:0:323618:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171907:0:323618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171914:0:323618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.171918:0:323618:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171925:0:323618:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171932:0:323618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171943:0:323618:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.171949:0:323618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.171955:0:323618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e012 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.171973:0:323618:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913441280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.171987:0:323618:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.172026:0:323618:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431467.172100:0:323618:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.172113:0:323618:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431467.172117:0:323618:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431467.172134:0:323618:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431467.172172:0:323618:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.172180:0:323618:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.172188:0:323618:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.172206:0:323618:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441408:192.168.203.128@tcp:101:tail.0 00000100:00100000:3.0:1789431467.172472:0:323618:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.177148:0:323618:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441408:192.168.203.128@tcp:101:tail.0 00010000:00010000:3.0:1789431467.177183:0:323618:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177198:0:323618:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177205:0:323618:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177213:0:323618:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177222:0:323618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177229:0:323618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.177233:0:323618:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177239:0:323618:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177246:0:323618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177253:0:323618:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.177259:0:323618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177266:0:323618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e019 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.177284:0:323618:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc5f80 x1876350913441408/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.177303:0:323618:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.177328:0:323618:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431467.177392:0:323618:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.177404:0:323618:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431467.177408:0:323618:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431467.177425:0:323618:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431467.177464:0:323618:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177470:0:323618:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.177479:0:323618:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.177497:0:323618:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441536:192.168.203.128@tcp:101:tail.0 00000100:00100000:3.0:1789431467.177782:0:323618:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.187655:0:323618:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441536:192.168.203.128@tcp:101:tail.0 00010000:00010000:3.0:1789431467.187771:0:323618:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187786:0:323618:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187794:0:323618:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187801:0:323618:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187811:0:323618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187817:0:323618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.187821:0:323618:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.187828:0:323618:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.188047:0:323618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.188055:0:323618:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.188062:0:323618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.188069:0:323618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e020 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.188088:0:323618:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc5f80 x1876350913441536/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.188103:0:323618:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.188144:0:323618:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431467.188270:0:323618:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.188286:0:323618:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62180, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431467.188293:0:323618:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431467.188310:0:323618:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431467.188357:0:323618:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.188366:0:323618:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431467.188376:0:323618:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431467.188383:0:323618:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5f80 x1876350913441792/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431467.188392:0:323618:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.188411:0:323618:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441792:192.168.203.128@tcp:101:tail.0 00000100:00100000:3.0:1789431467.188689:0:323618:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.196251:0:323618:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323618:1876350913441792:192.168.203.128@tcp:101:tail.0 00010000:00010000:3.0:1789431467.196276:0:323618:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196290:0:323618:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196297:0:323618:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196303:0:323618:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196312:0:323618:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196322:0:323618:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.196327:0:323618:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196333:0:323618:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196341:0:323618:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196348:0:323618:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.196353:0:323618:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.196360:0:323618:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e027 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323618 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.196378:0:323618:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc5f80 x1876350913441792/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.196393:0:323618:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.196419:0:323618:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431467.222987:0:323620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.223013:0:323620:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.223021:0:323620:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431467.223044:0:323620:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.223058:0:323620:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431467.223109:0:323620:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.223122:0:323620:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.223131:0:323620:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.223157:0:323620:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323620:1876350913441920:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431467.223246:0:323620:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.233168:0:323620:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323620:1876350913441920:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431467.233194:0:323620:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233208:0:323620:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233219:0:323620:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233230:0:323620:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233242:0:323620:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233251:0:323620:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233264:0:323620:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.233276:0:323620:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc5f80 x1876350913441920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431467.233291:0:323620:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233303:0:323620:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431467.233310:0:323620:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431467.233348:0:323620:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431467.233355:0:323620:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e02e 00000080:00010000:3.0:1789431467.233359:0:323620:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431467.233363:0:323620:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1c1bcb8 00010000:00010000:3.0:1789431467.233366:0:323620:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.233376:0:323620:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1789431467.271923:0:323620:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323620:1876350913443072:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431467.271946:0:323620:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.271968:0:323620:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.271979:0:323620:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.271986:0:323620:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.271995:0:323620:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.272001:0:323620:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.272006:0:323620:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.272023:0:323620:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.272030:0:323620:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.272036:0:323620:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.272043:0:323620:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.272048:0:323620:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e058 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323620 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.272065:0:323620:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdea00 x1876350913443072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.272078:0:323620:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.272104:0:323620:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431467.272116:0:323620:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:3.0:1789431467.272124:0:323620:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1c1b9a0 00010000:00010000:3.0:1789431467.272127:0:323620:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431467.272136:0:323620:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431467.280781:0:323621:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.280803:0:323621:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.280812:0:323621:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431467.280948:0:323621:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.281027:0:323621:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.281044:0:323621:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431467.281057:0:323621:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.281066:0:323621:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431467.281070:0:323621:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e02e 00000080:00010000:2.0:1789431467.281077:0:323621:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1c4bcb8 00010000:00010000:2.0:1789431467.281081:0:323621:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.281091:0:323621:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431467.281122:0:323621:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431467.281131:0:323621:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.281138:0:323621:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.281147:0:323621:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431467.281153:0:323621:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.281156:0:323621:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431467.281159:0:323621:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:2.0:1789431467.281162:0:323621:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1c4bb58 00010000:00010000:2.0:1789431467.281165:0:323621:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.281172:0:323621:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431467.281181:0:323621:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.281196:0:323621:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=9 00000080:00200000:2.0:1789431467.281200:0:323621:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:2.0:1789431467.281212:0:323621:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431467.281252:0:323621:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e05f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323621 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.281259:0:323621:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e05f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323621 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.281280:0:323621:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e05f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323621 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.281522:0:323621:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323621:1876350913443200:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431467.281681:0:323621:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000080:00200008:2.0:1789431467.288329:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.288338:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431467.288351:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.288364:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431467.288374:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431467.288381:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.288387:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431467.288390:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:2.0:1789431467.288396:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3cb8 00010000:00010000:2.0:1789431467.288400:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431467.288408:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431467.288429:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431467.288435:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.288441:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431467.288448:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431467.288455:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.288458:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431467.288460:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:2.0:1789431467.288463:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3b58 00010000:00010000:2.0:1789431467.288466:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431467.288474:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1789431467.288480:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.288486:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:2.0:1789431467.288491:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.288495:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431467.288504:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.288524:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1789431467.288532:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1789431467.288538:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.288541:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431467.288544:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e02e 00000080:00010000:2.0:1789431467.288547:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1ee3cb8 00010000:00010000:2.0:1789431467.288550:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1789431467.288556:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1789431467.288570:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431467.288577:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.288582:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1789431467.288589:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1789431467.288609:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.288613:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431467.288616:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:2.0:1789431467.288619:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3b58 00010000:00010000:2.0:1789431467.288622:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1789431467.288637:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1789431467.288645:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.288653:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=25 00000080:00200000:2.0:1789431467.288656:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.288678:0:323591:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=9 00000080:00200000:2.0:1789431467.288682:0:323591:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431467.288692:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_af59d8e2d7c08c0efa262b2135faa59b,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431467.288725:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.288733:0:323591:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323591 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.288743:0:323591:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323591 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.288767:0:323591:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323591:1876350913443456:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431467.288818:0:323591:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431467.294449:0:323591:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323591:1876350913443456:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431467.294470:0:323591:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294496:0:323591:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294504:0:323591:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294526:0:323591:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294536:0:323591:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294548:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294556:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294565:0:323591:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.294576:0:323591:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2100a80 x1876350913443456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431467.294587:0:323591:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294615:0:323591:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1672 (0 0) 00000080:00200000:2.0:1789431467.294621:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431467.294645:0:323591:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x688:0x0], updating i_size 4096 00000080:00200000:2.0:1789431467.294648:0:323591:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x688:0x0](ffff9811c4872a08): rc = 0 00000080:00010000:2.0:1789431467.294654:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e066 00000080:00010000:2.0:1789431467.294679:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.294683:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e066 from it ffffbc42c1ee3b20 00010000:00010000:2.0:1789431467.294687:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.294695:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1789431467.294727:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.294733:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.294752:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.294761:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.294771:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431467.294778:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.294782:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.294785:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e066 00000080:00010000:2.0:1789431467.294789:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e066 from it ffffbc42c1ee3cb8 00010000:00010000:2.0:1789431467.294792:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.294798:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e066 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab458 expref: -99 pid: 323591 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431467.295132:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431467.295144:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.295153:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431467.295163:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1789431467.295169:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.295173:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431467.295176:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:2.0:1789431467.295180:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3c18 00010000:00010000:2.0:1789431467.295184:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431467.295191:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1789431467.295199:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.295205:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=27 00000080:00200000:2.0:1789431467.295211:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.295216:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b, flags=11 00000080:00200000:2.0:1789431467.295223:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 101 00000080:00200000:2.0:1789431467.304773:0:323623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.304793:0:323623:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), flags=17 00000080:00200000:2.0:1789431467.304798:0:323623:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=0 00000002:00010000:2.0:1789431467.304823:0:323623:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431467.305496:0:323623:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e06d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323623 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.305518:0:323623:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e06d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323623 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.305528:0:323623:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e06d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323623 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.305555:0:323623:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323623:1876350913443584:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431467.305920:0:323623:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431467.377700:0:323626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913444864:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431467.377725:0:323626:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377738:0:323626:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377745:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377752:0:323626:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377763:0:323626:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377771:0:323626:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431467.377776:0:323626:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377783:0:323626:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377790:0:323626:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377797:0:323626:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431467.377803:0:323626:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.377810:0:323626:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27e09e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.377828:0:323626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bddf80 x1876350913444864/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431467.377883:0:323626:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x688:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431467.377919:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431467.377932:0:323626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431467.377941:0:323626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c63970 00010000:00010000:2.0:1789431467.377945:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431467.377958:0:323626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431467.378041:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.378049:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.378063:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.378067:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.378075:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.378085:0:323626:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), file ffff9811c3985080, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431467.378090:0:323626:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=open 00000002:00010000:2.0:1789431467.378111:0:323626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431467.378157:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.378164:0:323626:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431467.378175:0:323626:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431467.378180:0:323626:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913445120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431467.378189:0:323626:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.378209:0:323626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445120:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431467.378373:0:323626:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431467.382446:0:323626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445120:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431467.382470:0:323626:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382482:0:323626:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382489:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382496:0:323626:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382514:0:323626:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382521:0:323626:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431467.382525:0:323626:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382531:0:323626:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382538:0:323626:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382544:0:323626:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431467.382550:0:323626:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.382557:0:323626:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e0a5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.382581:0:323626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913445120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431467.382595:0:323626:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x688:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431467.382632:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431467.382645:0:323626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431467.382655:0:323626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c639a0 00010000:00010000:2.0:1789431467.382658:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431467.382673:0:323626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1789431467.385312:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385329:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.385344:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385348:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.385357:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385360:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.385370:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385373:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.385382:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385385:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.385392:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385396:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.385405:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385408:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.385417:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385420:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.385430:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.385442:0:323626:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), file ffff9811d0dcd380, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431467.385450:0:323626:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=open 00000002:00010000:2.0:1789431467.385477:0:323626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431467.385555:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.385565:0:323626:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431467.385576:0:323626:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431467.385587:0:323626:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913445376/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431467.385596:0:323626:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.385620:0:323626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445376:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431467.385807:0:323626:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431467.394001:0:323626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445376:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431467.394142:0:323626:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394155:0:323626:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394164:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394171:0:323626:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394179:0:323626:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394185:0:323626:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431467.394190:0:323626:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394196:0:323626:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394204:0:323626:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 5/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394212:0:323626:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431467.394218:0:323626:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.394224:0:323626:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0ac lrc: 0/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.394243:0:323626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913445376/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431467.394257:0:323626:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x688:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431467.394285:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431467.394298:0:323626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:2.0:1789431467.394307:0:323626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c639a0 00010000:00010000:2.0:1789431467.394311:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431467.394320:0:323626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431467.396234:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396247:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.396260:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396264:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.396271:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396274:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.396285:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396288:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.396297:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396301:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.396309:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396313:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.396322:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396325:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.396336:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396340:0:323626:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.396349:0:323626:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.396361:0:323626:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), file ffff9811e0087b00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431467.396369:0:323626:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=open 00000002:00010000:2.0:1789431467.396393:0:323626:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431467.396453:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.396463:0:323626:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431467.396475:0:323626:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431467.396486:0:323626:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913445632/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431467.396496:0:323626:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.396723:0:323626:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445632:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431467.397213:0:323626:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431467.402579:0:323626:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323626:1876350913445632:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431467.402606:0:323626:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402618:0:323626:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402635:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402642:0:323626:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402657:0:323626:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402671:0:323626:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431467.402675:0:323626:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402682:0:323626:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402689:0:323626:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402696:0:323626:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431467.402702:0:323626:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.402708:0:323626:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0b3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 323626 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.402727:0:323626:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913445632/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431467.402742:0:323626:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x688:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431467.402769:0:323626:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431467.402791:0:323626:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:2.0:1789431467.402806:0:323626:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c639a0 00010000:00010000:2.0:1789431467.402810:0:323626:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431467.402818:0:323626:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431467.413568:0:323629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.413591:0:323629:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.413598:0:323629:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.413634:0:323629:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.413653:0:323629:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.413677:0:323629:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431467.413687:0:323629:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.413695:0:323629:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.413699:0:323629:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.413707:0:323629:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1563cb8 00010000:00010000:2.0:1789431467.413710:0:323629:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431467.413718:0:323629:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431467.419445:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419461:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.419475:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419479:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.419489:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419492:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.419500:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419503:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.419511:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419514:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.419521:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419525:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.419533:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419536:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.419544:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419547:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.419555:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.419560:0:323628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.422255:0:323628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.422274:0:323628:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), flags=1 00000080:00200000:2.0:1789431467.422279:0:323628:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=getattr 00000002:00010000:2.0:1789431467.422302:0:323628:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_4a4fed3c2fc7765bcce912060752f4d4,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431467.422359:0:323628:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.422370:0:323628:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323628 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.422385:0:323628:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e0ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323628 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.422411:0:323628:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323628:1876350913445760:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431467.422517:0:323628:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000080:00200000:2.0:1789431467.472645:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.472676:0:323631:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.472684:0:323631:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.472706:0:323631:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.472724:0:323631:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431467.472738:0:323631:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431467.472745:0:323631:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.472752:0:323631:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.472756:0:323631:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.472763:0:323631:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c6bcb8 00010000:00010000:2.0:1789431467.472767:0:323631:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431467.472776:0:323631:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431467.476553:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476570:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.476584:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476588:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.476596:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476599:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.476609:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476612:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.476621:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476624:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.476635:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476638:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.476647:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476650:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.476658:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476674:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.476685:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.476690:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431467.477326:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477334:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.477344:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477347:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.477354:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477357:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.477365:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477368:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.477377:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477381:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.477390:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477392:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.477400:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477403:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.477411:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477414:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.477422:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477426:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.477650:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.477655:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.479207:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479220:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.479237:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479240:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.479247:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479250:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.479257:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479260:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.479269:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479272:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.479280:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479282:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.479291:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479294:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.479302:0:323631:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.479306:0:323631:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431467.498471:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.498489:0:323634:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.498498:0:323634:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:3.0:1789431467.499606:0:323634:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.499625:0:323634:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431467.499640:0:323634:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431467.499650:0:323634:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.499656:0:323634:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431467.499670:0:323634:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:3.0:1789431467.499677:0:323634:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c7bcb8 00010000:00010000:3.0:1789431467.499682:0:323634:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431467.499696:0:323634:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431467.504300:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504308:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.504320:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504324:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.504404:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504409:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.504419:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504422:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.504429:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504432:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.504439:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504442:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.504450:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504453:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.504464:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504467:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.504476:0:323634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.504480:0:323634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.530202:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530210:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530225:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530229:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530236:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530239:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530248:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530252:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431467.530451:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530457:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.530466:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530469:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530477:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530480:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530490:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530493:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530501:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530517:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530530:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530534:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530543:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530546:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530555:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530558:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530566:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530570:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.530730:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530735:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.530746:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530749:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530758:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530762:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530769:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530772:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530780:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530783:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.530791:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530794:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530802:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530805:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530813:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530816:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.530827:0:323636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.530867:0:323636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431467.543124:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.543144:0:323639:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.543153:0:323639:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:3.0:1789431467.543174:0:323639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.543191:0:323639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:3.0:1789431467.543212:0:323639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1789431467.543220:0:323639:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.543226:0:323639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431467.543230:0:323639:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:3.0:1789431467.543237:0:323639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c63cb8 00010000:00010000:3.0:1789431467.543241:0:323639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:3.0:1789431467.543249:0:323639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:3.0:1789431467.544539:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544552:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.544563:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544568:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.544578:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544581:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.544590:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544593:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.544602:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544605:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.544614:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544617:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.544626:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544628:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.544637:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544640:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.544648:0:323639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.544651:0:323639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00010000:00010000:2.0:1789431467.567984:0:323638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568002:0:323638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568013:0:323638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568025:0:323638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568043:0:323638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568050:0:323638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568057:0:323638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431467.568072:0:323638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdd500 x1876350913446528/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.65534' uid:65534 gid:0 projid:0 00000002:00010000:2.0:1789431467.568084:0:323638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568095:0:323638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1674 (0 0) 00000080:00200000:2.0:1789431467.568101:0:323638:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x689:0x0] suppgids 65534 -1: rc 0 00000080:00200000:2.0:1789431467.568121:0:323638:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x68a:0x0], updating i_size 4 00000080:00200000:2.0:1789431467.568124:0:323638:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x68a:0x0](ffff9811c48b5348): rc = 0 00000080:00010000:2.0:1789431467.568129:0:323638:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68a:0x0](ffff9811c48b5348) for lock 0x1ffb2ee1bd27e0c8 00000080:00010000:2.0:1789431467.568139:0:323638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68a:0x0](ffff9811c48b5348) 00000080:00010000:2.0:1789431467.568143:0:323638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e0c8 from it ffffbc42c1bfbe00 00010000:00010000:2.0:1789431467.568146:0:323638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.568153:0:323638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431467.568188:0:323638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431467.568196:0:323638:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8) 00010000:00010000:2.0:1789431467.568225:0:323638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.568266:0:323638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.568273:0:323638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431467.568276:0:323638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27e0c8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab5b6 expref: -99 pid: 323638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1789431467.568300:0:323638:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431467.568307:0:323638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102d80 x1876350913446784/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431467.568333:0:323638:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323638:1876350913446784:192.168.203.128@tcp:36:pjdfstest.65534 00000100:00100000:2.0:1789431467.568433:0:323638:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.572151:0:323638:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323638:1876350913446784:192.168.203.128@tcp:36:pjdfstest.65534 00000080:00200000:3.0:1789431467.572201:0:323638:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x68a:0x0](ffff9811c48b5348) 00000080:00200000:2.0:1789431467.584091:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.584113:0:323641:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.584121:0:323641:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.584143:0:323641:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.584160:0:323641:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431467.584175:0:323641:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431467.584183:0:323641:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.584190:0:323641:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.584195:0:323641:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.584201:0:323641:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c163fcb8 00010000:00010000:2.0:1789431467.584206:0:323641:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431467.584215:0:323641:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1789431467.589061:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589077:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.589089:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589094:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589103:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589106:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589115:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589119:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589128:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589131:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589252:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589259:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589270:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589273:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589284:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589287:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589297:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589301:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431467.589496:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589501:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.589592:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589598:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589609:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589613:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589697:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589703:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589717:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589721:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.589730:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589733:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589742:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589745:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589754:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589757:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.589768:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.589771:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.591373:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591382:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.591395:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591399:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.591409:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591413:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.591421:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591424:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.591431:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591434:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.591442:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591445:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.591453:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591457:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.591466:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591469:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.591479:0:323641:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.591483:0:323641:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431467.617399:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617415:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.617428:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617433:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.617441:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617444:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.617452:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617455:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.617463:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617466:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.617474:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617477:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.617485:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617488:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.617495:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617498:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.617505:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.617509:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.618292:0:323643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.618301:0:323643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, flags=1 00000080:00200008:2.0:1789431467.618316:0:323643:0:(llite_lib.c:2428:ll_setattr_raw()) START file pjdfstest_4a4fed3c2fc7765bcce912060752f4d4:[0x200000402:0x689:0x0](ffff9811c48700c8) current size 4096, valid attrs 41, mode 41a4, uid -1053089792, gid -26607, new size 0, atime 1789431467.000000000, mtime 1789431467.000000000, ctime 1789431467.000000000 00010000:00010000:2.0:1789431467.618347:0:323643:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27e0c1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab58c expref: -99 pid: 323633 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.618405:0:323643:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27e0c1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab58c expref: -99 pid: 323633 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431467.618412:0:323643:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431467.618415:0:323643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ea00/0x1ffb2ee1bd27e0c1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab58c expref: -99 pid: 323633 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1789431467.618436:0:323643:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431467.618449:0:323643:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102a00 x1876350913447040/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431467.618472:0:323643:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323643:1876350913447040:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431467.618569:0:323643:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000080:00200000:2.0:1789431467.639482:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.639501:0:323646:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.639509:0:323646:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.639530:0:323646:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.639548:0:323646:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431467.639561:0:323646:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431467.639569:0:323646:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.639575:0:323646:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.639579:0:323646:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.639585:0:323646:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1cb3cb8 00010000:00010000:2.0:1789431467.639588:0:323646:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431467.639596:0:323646:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1789431467.642330:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642343:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.642356:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642359:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642367:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642370:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642378:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642381:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642391:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642394:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642403:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642407:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642415:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642419:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642428:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642431:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642439:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642443:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431467.642685:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642691:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.642703:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642707:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642716:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642720:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642727:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642730:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642737:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642739:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.642747:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642750:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642759:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642762:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642770:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642773:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.642781:0:323646:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.642785:0:323646:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.669166:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.669187:0:323649:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.669196:0:323649:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:3.0:1789431467.669217:0:323649:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.669246:0:323649:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1789431467.669262:0:323649:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1789431467.669270:0:323649:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.669278:0:323649:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431467.669281:0:323649:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:3.0:1789431467.669288:0:323649:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c53cb8 00010000:00010000:3.0:1789431467.669291:0:323649:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1789431467.669299:0:323649:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1789431467.670570:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670581:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.670592:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670596:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.670602:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670605:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.670611:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670614:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.670620:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670623:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.670629:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670632:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.670640:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670642:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.670649:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670652:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.670671:0:323649:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.670674:0:323649:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.674557:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674575:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.674587:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674591:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.674599:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674603:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.674613:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674616:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.674625:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674628:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.674636:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674639:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.674646:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674648:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.674655:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674658:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.674679:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.674683:0:323648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.675715:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.675730:0:323648:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), flags=16 00000080:00200000:2.0:1789431467.675734:0:323648:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=0 00000002:00010000:2.0:1789431467.675757:0:323648:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_4a4fed3c2fc7765bcce912060752f4d4,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431467.675806:0:323648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323648 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.675816:0:323648:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323648 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.675825:0:323648:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323648 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.675993:0:323648:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323648:1876350913447168:192.168.203.128@tcp:101:pjdfstest.65534 00000100:00100000:2.0:1789431467.676077:0:323648:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.679576:0:323648:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323648:1876350913447168:192.168.203.128@tcp:101:pjdfstest.65534 00010000:00010000:3.0:1789431467.679599:0:323648:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679615:0:323648:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679620:0:323648:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679637:0:323648:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679649:0:323648:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679657:0:323648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679676:0:323648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679683:0:323648:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.679695:0:323648:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d2100e00 x1876350913447168/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.65534' uid:65534 gid:0 projid:0 00000002:00010000:3.0:1789431467.679709:0:323648:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679719:0:323648:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1673 (0 0) 00000080:00200000:3.0:1789431467.679725:0:323648:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x688:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1789431467.679750:0:323648:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x689:0x0], updating i_size 4096 00000080:00200000:3.0:1789431467.679753:0:323648:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x689:0x0](ffff9811c48700c8): rc = 0 00000080:00010000:3.0:1789431467.679758:0:323648:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x689:0x0](ffff9811c48700c8) for lock 0x1ffb2ee1bd27e0cf 00000080:00010000:3.0:1789431467.679770:0:323648:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x689:0x0](ffff9811c48700c8) 00000080:00010000:3.0:1789431467.679773:0:323648:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e0cf from it ffffbc42c1c93ab0 00010000:00010000:3.0:1789431467.679776:0:323648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.679790:0:323648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e0cf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab5ee expref: -99 pid: 323648 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431467.679822:0:323648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41a4 mask 1 00000080:00200000:2.0:1789431467.686044:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.686064:0:323651:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.686072:0:323651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.686093:0:323651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.686111:0:323651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431467.686125:0:323651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1789431467.686131:0:323651:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.686138:0:323651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.686142:0:323651:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.686151:0:323651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1cbbcb8 00010000:00010000:2.0:1789431467.686155:0:323651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431467.686163:0:323651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1789431467.689312:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689325:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.689338:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689342:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689350:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689354:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689361:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689363:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689370:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689373:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689381:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689384:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689392:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689395:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689402:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689405:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689413:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689416:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431467.689601:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689607:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.689615:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689618:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689641:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689644:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689654:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689656:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689789:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689794:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.689806:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689809:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689817:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689819:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689827:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689829:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.689869:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.689873:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.690508:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690513:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.690520:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690522:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.690526:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690529:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.690536:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690539:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.690546:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690548:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.690556:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690559:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.690565:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690568:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.690574:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690577:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.690584:0:323651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.690588:0:323651:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431467.697495:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.697512:0:323654:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.697521:0:323654:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.697542:0:323654:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.697560:0:323654:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1789431467.697574:0:323654:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1789431467.697583:0:323654:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.697591:0:323654:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.697725:0:323654:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.697739:0:323654:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1cc3cb8 00010000:00010000:2.0:1789431467.697746:0:323654:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1789431467.697760:0:323654:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:3.0:1789431467.700093:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700104:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.700117:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700121:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.700129:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700131:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.700140:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700142:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.700149:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700152:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.700159:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700162:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.700168:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700171:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.700179:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700182:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.700189:0:323654:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.700193:0:323654:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.709566:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.709583:0:323656:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.709592:0:323656:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:3.0:1789431467.709612:0:323656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.709630:0:323656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:3.0:1789431467.709645:0:323656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:3.0:1789431467.709653:0:323656:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.709673:0:323656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431467.709678:0:323656:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:3.0:1789431467.709685:0:323656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c03cb8 00010000:00010000:3.0:1789431467.709689:0:323656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:3.0:1789431467.709699:0:323656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:3.0:1789431467.710978:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.710989:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.711003:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711007:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711015:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711018:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711026:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711029:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711037:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711040:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711049:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711052:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711060:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711063:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711071:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711073:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711082:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711089:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431467.711259:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711264:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.711273:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711276:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711287:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711290:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711298:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711301:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711309:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711312:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711318:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711322:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711330:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711333:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711342:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711345:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711353:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711358:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.711484:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711489:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.711499:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711502:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711509:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711512:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711520:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711522:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711531:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711534:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.711542:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711544:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711552:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711555:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711563:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711565:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.711572:0:323656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.711575:0:323656:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431467.728848:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.728868:0:323659:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.728875:0:323659:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.728897:0:323659:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.728914:0:323659:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431467.728929:0:323659:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1789431467.728936:0:323659:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x688:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431467.728943:0:323659:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431467.728947:0:323659:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x688:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e089 00000080:00010000:2.0:1789431467.728954:0:323659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e089 from it ffffbc42c1c13cb8 00010000:00010000:2.0:1789431467.728958:0:323659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431467.728966:0:323659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e089 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab4cf expref: -99 pid: 323626 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1789431467.730255:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730265:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.730275:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730278:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.730285:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730288:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.730295:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730298:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.730306:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730309:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.730315:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730319:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.730326:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730329:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.730336:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730339:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.730347:0:323659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.730351:0:323659:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.733215:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733231:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.733243:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733248:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.733258:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733261:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.733270:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733273:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.733281:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733284:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.733292:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733295:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.733304:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733307:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.733315:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733318:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.733325:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.733329:0:323658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.734083:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.734099:0:323658:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), flags=16 00000080:00200000:3.0:1789431467.734104:0:323658:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, dir=[0x200000402:0x688:0x0](ffff9811c4872a08), intent=0 00000002:00010000:3.0:1789431467.734129:0:323658:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_4a4fed3c2fc7765bcce912060752f4d4,[0x0:0x0:0x0]) in obj [0x200000402:0x688:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431467.734181:0:323658:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323658 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.734191:0:323658:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323658 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.734202:0:323658:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323658 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.734226:0:323658:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323658:1876350913447552:192.168.203.128@tcp:101:pjdfstest.65534 00000100:00100000:3.0:1789431467.734320:0:323658:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.741348:0:323658:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323658:1876350913447552:192.168.203.128@tcp:101:pjdfstest.65534 00010000:00010000:3.0:1789431467.741375:0:323658:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741396:0:323658:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741404:0:323658:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741415:0:323658:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741426:0:323658:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741437:0:323658:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741446:0:323658:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741453:0:323658:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.741468:0:323658:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc5f80 x1876350913447552/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.65534' uid:65534 gid:0 projid:0 00000002:00010000:3.0:1789431467.741543:0:323658:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741554:0:323658:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1673 (0 0) 00000080:00200000:3.0:1789431467.741560:0:323658:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x688:0x0] suppgids -1 -1: rc 0 00000080:00200000:3.0:1789431467.741582:0:323658:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x689:0x0], updating i_size 4096 00000080:00200000:3.0:1789431467.741585:0:323658:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x689:0x0](ffff9811c48700c8): rc = 0 00000080:00010000:3.0:1789431467.741591:0:323658:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x689:0x0](ffff9811c48700c8) for lock 0x1ffb2ee1bd27e0d6 00000080:00010000:3.0:1789431467.741603:0:323658:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x689:0x0](ffff9811c48700c8) 00000080:00010000:3.0:1789431467.741607:0:323658:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e0d6 from it ffffbc42c1c93ab0 00010000:00010000:3.0:1789431467.741609:0:323658:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.741617:0:323658:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e0d6 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x689:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab611 expref: -99 pid: 323658 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431467.741648:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.741746:0:323658:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8), flags=1536 00000080:00200000:3.0:1789431467.741750:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431467.741757:0:323658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431467.741761:0:323658:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8), target=test 00000020:00100000:3.0:1789431467.741873:0:323658:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431467.741881:0:323658:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5f80 x1876350913447808/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431467.741903:0:323658:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323658:1876350913447808:192.168.203.128@tcp:36:pjdfstest.65534 00000100:00100000:3.0:1789431467.741971:0:323658:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecb40 going to sleep for 11 seconds 00000080:00200000:2.0:1789431467.756316:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756329:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.756342:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756346:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756355:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756358:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756366:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756369:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756376:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756379:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756386:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756389:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756396:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756399:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756406:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756409:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756415:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756419:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431467.756587:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756592:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.756602:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756605:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756614:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756616:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756625:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756629:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756638:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756641:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756650:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756653:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756676:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756679:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756689:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756693:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756701:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756705:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431467.756876:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756881:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.756891:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756894:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756904:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756907:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756915:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756918:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756928:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756931:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.756939:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756943:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756951:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756954:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756962:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756965:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.756974:0:323661:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.756978:0:323661:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431467.771173:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771189:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431467.771203:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771207:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.771215:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771219:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.771226:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771229:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.771237:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771240:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431467.771249:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771252:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.771260:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771264:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.771272:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771274:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431467.771281:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.771284:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431467.773194:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.773206:0:323663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_4a4fed3c2fc7765bcce912060752f4d4, flags=16 00000080:00200000:3.0:1789431467.773213:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.773226:0:323663:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8), flags=0 00000080:00200000:3.0:1789431467.773230:0:323663:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8), intent=getattr 00000002:00010000:3.0:1789431467.773255:0:323663:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_ba19094b855827abf55af892a99e2915,[0x0:0x0:0x0]) in obj [0x200000402:0x689:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431467.773310:0:323663:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x689:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 323663 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.773321:0:323663:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x689:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323663 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.773330:0:323663:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x689:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 323663 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.773354:0:323663:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323663:1876350913447936:192.168.203.128@tcp:101:pjdfstest.65534 00000100:00100000:3.0:1789431467.773443:0:323663:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cece40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.783267:0:323663:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323663:1876350913447936:192.168.203.128@tcp:101:pjdfstest.65534 00010000:00010000:3.0:1789431467.783291:0:323663:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x689:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783311:0:323663:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: --/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783320:0:323663:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: --/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783331:0:323663:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783340:0:323663:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783353:0:323663:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783365:0:323663:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783372:0:323663:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.783383:0:323663:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc5180 x1876350913447936/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.65534' uid:65534 gid:0 projid:0 00000002:00010000:3.0:1789431467.783395:0:323663:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783405:0:323663:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1675 (0 0) 00000080:00200000:3.0:1789431467.783411:0:323663:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x689:0x0] suppgids 65534 -1: rc 0 00000080:00200000:3.0:1789431467.783429:0:323663:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x68b:0x0], updating i_size 4 00000080:00200000:3.0:1789431467.783432:0:323663:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x68b:0x0](ffff9811c48b4b08): rc = 0 00000080:00010000:3.0:1789431467.783437:0:323663:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68b:0x0](ffff9811c48b4b08) for lock 0x1ffb2ee1bd27e0dd 00000080:00010000:3.0:1789431467.783445:0:323663:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68b:0x0](ffff9811c48b4b08) 00000080:00010000:3.0:1789431467.783448:0:323663:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e0dd from it ffffbc42c1c13e00 00010000:00010000:3.0:1789431467.783452:0:323663:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.783461:0:323663:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431467.783487:0:323663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x689:0x0](ffff9811c48700c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431467.783495:0:323663:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_ba19094b855827abf55af892a99e2915, dir=[0x200000402:0x689:0x0](ffff9811c48700c8) 00010000:00010000:3.0:1789431467.783522:0:323663:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431467.783560:0:323663:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431467.783567:0:323663:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431467.783570:0:323663:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e0dd lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68b:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab63b expref: -99 pid: 323663 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1789431467.783588:0:323663:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431467.783594:0:323663:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5180 x1876350913448192/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431467.783614:0:323663:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323663:1876350913448192:192.168.203.128@tcp:36:pjdfstest.65534 00000100:00100000:3.0:1789431467.783704:0:323663:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cece40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.788354:0:323663:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323663:1876350913448192:192.168.203.128@tcp:36:pjdfstest.65534 00000080:00200000:3.0:1789431467.788403:0:323663:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x68b:0x0](ffff9811c48b4b08) 00000080:00200000:2.0:1789431467.810350:0:323669:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431467.810368:0:323669:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_af59d8e2d7c08c0efa262b2135faa59b:[0x200000402:0x688:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431467.810376:0:323669:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x688:0x0](ffff9811c4872a08),name=pjdfstest_af59d8e2d7c08c0efa262b2135faa59b 00000002:00010000:2.0:1789431467.810395:0:323669:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x688:0x0]) in obj [0x200000402:0x688:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431467.810449:0:323669:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e0e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 323669 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.810460:0:323669:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e0e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 323669 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.810469:0:323669:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e0e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x688:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 323669 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.810492:0:323669:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323669:1876350913448576:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431467.811154:0:323669:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000080:00200008:3.0:1789431467.858073:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.858083:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431467.858101:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.858117:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431467.858130:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1789431467.858137:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.858144:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431467.858147:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431467.858154:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431467.858158:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431467.858165:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431467.858187:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431467.858193:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.858199:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431467.858207:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431467.858212:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.858216:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431467.858219:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431467.858222:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3b58 00010000:00010000:3.0:1789431467.858225:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431467.858231:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1789431467.858240:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.858246:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431467.858253:0:323591:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.858257:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431467.858267:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.858275:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431467.858282:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1789431467.858289:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.858292:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431467.858295:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e02e 00000080:00010000:3.0:1789431467.858298:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1ee3cb8 00010000:00010000:3.0:1789431467.858300:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:3.0:1789431467.858307:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431467.858322:0:323591:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431467.858328:0:323591:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.858333:0:323591:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431467.858341:0:323591:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1789431467.858346:0:323591:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.858350:0:323591:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431467.858352:0:323591:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27df63 00000080:00010000:3.0:1789431467.858355:0:323591:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27df63 from it ffffbc42c1ee3c18 00010000:00010000:3.0:1789431467.858358:0:323591:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431467.858365:0:323591:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27df63 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab133 expref: -99 pid: 323077 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1789431467.858372:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.858375:0:323591:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:3.0:1789431467.858379:0:323591:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:3.0:1789431467.860441:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431467.860459:0:323674:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431467.860466:0:323674:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431467.860485:0:323674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431467.860501:0:323674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431467.860515:0:323674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1789431467.860521:0:323674:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431467.860528:0:323674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431467.860531:0:323674:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e02e 00000080:00010000:3.0:1789431467.860536:0:323674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e02e from it ffffbc42c1cabcb8 00010000:00010000:3.0:1789431467.860539:0:323674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1789431467.860552:0:323674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e02e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab3a2 expref: -99 pid: 323620 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431467.865589:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865603:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431467.865615:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865619:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.865627:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865630:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.865639:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865641:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.865648:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865650:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431467.865728:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865734:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.865750:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865753:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.865761:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865764:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431467.865773:0:323674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431467.865778:0:323674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00010000:00010000:2.0:1789431467.871025:0:6160:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1789431467.871032:0:6160:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27e115 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab70d expref: -99 pid: 323673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.871040:0:6160:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c0030e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431467.871044:0:6160:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27e115 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab70d expref: -99 pid: 323673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.871052:0:6160:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27e115 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab70d expref: -99 pid: 323673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.871063:0:6160:0:(ldlm_request.c:1774:ldlm_cli_cancel()) ### lock is already being destroyed 00010000:00010000:2.0:1789431467.871066:0:6160:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030e00/0x1ffb2ee1bd27e115 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x688:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab70d expref: -99 pid: 323673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.871074:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431467.871076:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431467.871077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789431467.924023:0:323676:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431467.924033:0:323676:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431467.924061:0:323676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431467.924122:0:323676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.924133:0:323676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431467.924142:0:323676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431467.924168:0:323676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323676:1876350913450368:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431467.924286:0:323676:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:3.0:1789431467.932186:0:323676:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323676:1876350913450368:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431467.932210:0:323676:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932224:0:323676:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932231:0:323676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932239:0:323676:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932247:0:323676:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932253:0:323676:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431467.932257:0:323676:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932264:0:323676:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932271:0:323676:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932281:0:323676:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431467.932287:0:323676:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932294:0:323676:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e123 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431467.932311:0:323676:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2103480 x1876350913450368/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431483 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431467.932325:0:323676:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431467.932351:0:323676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431467.932363:0:323676:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431467.932372:0:323676:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1d2b970 00010000:00010000:3.0:1789431467.932375:0:323676:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431467.932383:0:323676:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431467.932451:0:323676:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431467.932464:0:323676:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431467.932468:0:323676:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431467.932485:0:323676:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431467.932524:0:323676:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e12a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932532:0:323676:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e12a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431467.932541:0:323676:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e12a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 323676 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431467.932561:0:323676:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:323676:1876350913450496:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431467.934282:0:323676:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cece40 going to sleep for 11 seconds 00000001:02000400:3.0:1789431469.308223:0:323808:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:2.0:1789431470.758031:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431473.831032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431476.902071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431477.247813:0:324525:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_06: symlink returns EACCES if the parent directory of the file to be created denies write permission ========================================================== 20:17:55 (1789431475) 00010000:00010000:2.0:1789431478.950013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431482.022974:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431482.051290:0:6160:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1789431482.051301:0:6160:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051315:0:6160:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c0030200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431482.051322:0:6160:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051331:0:6160:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051354:0:6160:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051373:0:6160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431482.051384:0:6160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431482.051420:0:6160:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 2/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051429:0:6160:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431482.051456:0:6160:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.051470:0:6160:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030200/0x1ffb2ee1bd27e14d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab7ca expref: -99 pid: 324901 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431482.051487:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913452800:192.168.203.128@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1789431482.051502:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431482.051506:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431482.051507:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431482.058008:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913452800:192.168.203.128@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1789431482.060635:0:324901:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324901:1876350913453056:192.168.203.128@tcp:36:lfs.0 00000080:00200008:2.0:1789431482.060724:0:324901:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200000:2.0:1789431482.154380:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154396:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431482.154408:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154412:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431482.154420:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154423:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431482.154431:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154434:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431482.154441:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154444:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431482.154453:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154457:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431482.154465:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154468:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431482.154475:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154478:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431482.154486:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.154495:0:324905:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d8d58300, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431482.154502:0:324905:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431482.154523:0:324905:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431482.154599:0:324905:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e15b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.154609:0:324905:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e15b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431482.154619:0:324905:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431482.154631:0:324905:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913453440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431482.154641:0:324905:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e15b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431482.154662:0:324905:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324905:1876350913453440:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431482.156980:0:324905:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000080:00200000:3.0:1789431482.513828:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514114:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431482.514132:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514137:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431482.514148:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514151:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431482.514159:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514162:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431482.514169:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514172:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431482.514179:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514182:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431482.514190:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514193:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431482.514200:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514203:0:324905:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431482.514211:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431482.514223:0:324905:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0c26300, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431482.514230:0:324905:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431482.514265:0:324905:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: librt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431482.514329:0:324905:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.514340:0:324905:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431482.514352:0:324905:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431482.514365:0:324905:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4000 x1876350913454976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431482.514375:0:324905:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431482.514396:0:324905:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324905:1876350913454976:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431482.514630:0:324905:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1a7dc00 going to sleep for 11 seconds 00000100:00100000:3.0:1789431482.529145:0:324905:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324905:1876350913454976:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431482.529169:0:324905:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529181:0:324905:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529187:0:324905:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529194:0:324905:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529202:0:324905:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529208:0:324905:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431482.529212:0:324905:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529219:0:324905:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529225:0:324905:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529231:0:324905:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431482.529236:0:324905:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431482.529242:0:324905:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e185 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431482.529258:0:324905:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4000 x1876350913454976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431498 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431482.529273:0:324905:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431482.529303:0:324905:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431482.529317:0:324905:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00010000:3.0:1789431482.529327:0:324905:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c15779a0 00010000:00010000:3.0:1789431482.529332:0:324905:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431482.529346:0:324905:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1789431482.714289:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.714303:0:324905:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431482.714319:0:324905:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431482.714324:0:324905:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431482.714349:0:324905:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: testrules.yml,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431482.714407:0:324905:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e19a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.714418:0:324905:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e19a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431482.714428:0:324905:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e19a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324905 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431482.714451:0:324905:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324905:1876350913455360:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431482.714542:0:324905:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9900 going to sleep for 11 seconds 00000080:00200000:3.0:1789431483.016378:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.016401:0:324908:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.016409:0:324908:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.016518:0:324908:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.016538:0:324908:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:3.0:1789431483.016564:0:324908:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:3.0:1789431483.016574:0:324908:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.016582:0:324908:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.016585:0:324908:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.016593:0:324908:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1c53cb8 00010000:00010000:3.0:1789431483.016597:0:324908:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:3.0:1789431483.016605:0:324908:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:3.0:1789431483.018330:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018345:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.018358:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018363:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.018373:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018376:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.018383:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018386:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.018394:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018397:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.018404:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018408:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.018417:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018420:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.018439:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018443:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.018454:0:324908:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.018458:0:324908:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.034742:0:324911:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.034762:0:324911:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.034769:0:324911:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.034797:0:324911:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.034829:0:324911:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431483.035066:0:324911:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:3.0:1789431483.035074:0:324911:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.035081:0:324911:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.035085:0:324911:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.035092:0:324911:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c163fcb8 00010000:00010000:3.0:1789431483.035096:0:324911:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431483.035105:0:324911:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1789431483.048512:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789431483.068212:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.068234:0:324914:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.068242:0:324914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.068268:0:324914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.068288:0:324914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1789431483.068303:0:324914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1789431483.068312:0:324914:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.068319:0:324914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.068322:0:324914:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.068329:0:324914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1c53cb8 00010000:00010000:3.0:1789431483.068333:0:324914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1789431483.068343:0:324914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:3.0:1789431483.069617:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069629:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.069640:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069644:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.069651:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069654:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.069661:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069663:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.069671:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069674:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.069681:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069685:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.069693:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069696:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.069704:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069707:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.069713:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.069717:0:324914:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.070620:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.070631:0:324914:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.070637:0:324914:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.070660:0:324914:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.070677:0:324914:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:3.0:1789431483.070691:0:324914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:3.0:1789431483.070699:0:324914:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.070706:0:324914:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.070710:0:324914:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.070717:0:324914:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1c53cb8 00010000:00010000:3.0:1789431483.070720:0:324914:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:3.0:1789431483.070729:0:324914:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1789431483.070752:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.070756:0:324914:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:3.0:1789431483.070761:0:324914:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:3.0:1789431483.070778:0:324914:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:3.0:1789431483.070790:0:324914:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:3.0:1789431483.071550:0:324914:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.074498:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.074518:0:324916:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.074526:0:324916:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431483.074564:0:324916:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.074583:0:324916:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1789431483.074596:0:324916:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:2.0:1789431483.074604:0:324916:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.074612:0:324916:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.074615:0:324916:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:2.0:1789431483.074622:0:324916:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1677cb8 00010000:00010000:2.0:1789431483.074625:0:324916:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:2.0:1789431483.074633:0:324916:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:2.0:1789431483.079781:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079796:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.079818:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079823:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.079854:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079858:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.079868:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079872:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.079881:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079884:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.079892:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079896:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.079915:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079919:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.079928:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.079931:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.080490:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.080504:0:324916:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0086a80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431483.080512:0:324916:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431483.080535:0:324916:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libsigsegv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431483.080603:0:324916:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.080613:0:324916:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431483.080623:0:324916:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431483.080634:0:324916:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103b80 x1876350913456000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431483.080644:0:324916:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.080664:0:324916:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324916:1876350913456000:192.168.203.128@tcp:101:awk.0 00000100:00100000:2.0:1789431483.082044:0:324916:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431483.088268:0:324916:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324916:1876350913456000:192.168.203.128@tcp:101:awk.0 00010000:00010000:3.0:1789431483.088298:0:324916:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088314:0:324916:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088322:0:324916:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088331:0:324916:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088341:0:324916:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088349:0:324916:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431483.088354:0:324916:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088362:0:324916:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088370:0:324916:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088378:0:324916:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431483.088385:0:324916:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.088393:0:324916:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e1af lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431483.088418:0:324916:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2103b80 x1876350913456000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431483.088436:0:324916:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431483.088473:0:324916:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:3.0:1789431483.088489:0:324916:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00010000:3.0:1789431483.088500:0:324916:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c16779a0 00010000:00010000:3.0:1789431483.088504:0:324916:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:3.0:1789431483.088518:0:324916:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:3.0:1789431483.089392:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089408:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.089421:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089426:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.089439:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089442:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.089451:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089453:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.089460:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089463:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.089471:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089475:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.089483:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089486:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.089494:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.089497:0:324916:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.089504:0:324916:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.091335:0:324916:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8195680, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431483.098653:0:324916:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431483.098797:0:324916:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libreadline.so.7,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431483.099077:0:324916:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e1b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.099088:0:324916:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e1b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431483.099101:0:324916:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431483.099115:0:324916:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101880 x1876350913456384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431483.099126:0:324916:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e1b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324916 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.099152:0:324916:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324916:1876350913456384:192.168.203.128@tcp:101:awk.0 00000100:00100000:2.0:1789431483.099402:0:324916:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431483.162865:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.162885:0:324921:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.162893:0:324921:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431483.162923:0:324921:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.162940:0:324921:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1789431483.162955:0:324921:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:2.0:1789431483.162962:0:324921:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.162969:0:324921:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.162973:0:324921:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:2.0:1789431483.162980:0:324921:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1753cb8 00010000:00010000:2.0:1789431483.162984:0:324921:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:2.0:1789431483.162993:0:324921:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1789431483.165450:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.165465:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.165476:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.165480:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.165489:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.165492:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.165500:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.165503:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.165513:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.165515:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.165517:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200008:3.0:1789431483.165528:0:324919:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.165529:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.165538:0:324919:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.165582:0:324919:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.165598:0:324919:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:3.0:1789431483.165620:0:324919:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:3.0:1789431483.165629:0:324919:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.165637:0:324919:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.165641:0:324919:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.165647:0:324919:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1d53cb8 00010000:00010000:3.0:1789431483.165652:0:324919:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:3.0:1789431483.165661:0:324919:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:3.0:1789431483.167339:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167351:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.167363:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167366:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.167373:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167376:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.167383:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167386:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.167394:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167396:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.167403:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167407:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.167417:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167420:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.167429:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167432:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.167440:0:324919:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.167443:0:324919:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.167523:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.167662:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.167668:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.169937:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.169949:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.169961:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.169964:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431483.170173:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170179:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.170187:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170190:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.170260:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170264:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.170275:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170277:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.170284:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170287:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.170480:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170485:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.170493:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170496:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.170503:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170506:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.170512:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170515:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431483.170962:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170968:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.170980:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.170984:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.170999:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171003:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.171014:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171018:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.171050:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171055:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.171066:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171069:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.171077:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171079:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.171088:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171090:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.171097:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171101:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431483.171955:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171964:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.171975:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171979:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.171992:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.171996:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172013:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172017:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172033:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172036:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172044:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172047:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172279:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172285:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172295:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172298:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172306:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172310:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431483.172519:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172524:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.172532:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172611:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172624:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172627:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172635:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172637:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172644:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172646:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.172653:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172656:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172663:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172666:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172674:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172676:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.172683:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.172687:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431483.172999:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173004:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.173013:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173016:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173024:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173027:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173034:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173037:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173046:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173048:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173056:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173059:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173067:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173070:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173076:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173079:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173085:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173088:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431483.173272:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173278:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.173290:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173293:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173301:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173304:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173312:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173316:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173325:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173329:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173337:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173340:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173350:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173353:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173364:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173368:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173377:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173381:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.173901:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173908:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.173920:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173924:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173934:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173937:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173946:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173949:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173959:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173962:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.173972:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173975:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.173987:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.173992:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.174004:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.174008:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.174018:0:324921:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.174022:0:324921:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431483.228506:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.228526:0:324924:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.228534:0:324924:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.228575:0:324924:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.228596:0:324924:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431483.228612:0:324924:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:3.0:1789431483.228628:0:324924:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.228637:0:324924:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.228640:0:324924:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.228647:0:324924:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c178bcb8 00010000:00010000:3.0:1789431483.228650:0:324924:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431483.228660:0:324924:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:3.0:1789431483.230543:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230566:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.230580:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230583:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.230593:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230597:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.230605:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230608:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.230615:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230619:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.230627:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230631:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.230641:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230643:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.230650:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230654:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.230663:0:324924:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.230667:0:324924:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.279196:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.279216:0:324929:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.279223:0:324929:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.279253:0:324929:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.279272:0:324929:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:3.0:1789431483.279286:0:324929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:3.0:1789431483.279294:0:324929:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.279302:0:324929:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431483.279305:0:324929:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:3.0:1789431483.279312:0:324929:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c1bebcb8 00010000:00010000:3.0:1789431483.279316:0:324929:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:3.0:1789431483.279329:0:324929:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1789431483.284351:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284359:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.284371:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284376:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.284390:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284395:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.284408:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284411:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.284422:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284425:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.284435:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284438:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.284448:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284451:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.284468:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284471:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.284489:0:324929:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.284494:0:324929:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.295638:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295661:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.295681:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295690:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.295702:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295706:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.295715:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295718:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.295732:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295736:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.295748:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295754:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.295769:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295772:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.295789:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295793:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.295810:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.295816:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431483.296028:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296033:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.296042:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296045:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296054:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296057:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296065:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296068:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296077:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296080:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296089:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296092:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296100:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296102:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296109:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296111:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296118:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296122:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431483.296487:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296493:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.296504:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296508:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296517:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296521:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296528:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296531:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296539:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296543:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296551:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296565:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296575:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296578:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296587:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296590:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296598:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296603:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431483.296747:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296752:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.296763:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296766:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296774:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296777:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296786:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296789:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296797:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296800:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.296809:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296812:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296820:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296823:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296886:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296890:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.296901:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.296905:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431483.297038:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297043:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.297052:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297055:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.297064:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297067:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.297079:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297083:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.297092:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297095:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.297103:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297106:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.297115:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297118:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.297127:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.297130:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298077:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298093:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431483.298407:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298414:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.298426:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298430:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298440:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298442:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298450:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298453:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298460:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298462:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298470:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298474:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298482:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298485:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298493:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298496:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298507:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298511:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431483.298658:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298663:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.298672:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298675:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298683:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298686:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298694:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298697:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298705:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298708:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298717:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298720:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298730:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298733:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298741:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298744:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.298752:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298756:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.298963:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298968:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.298978:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298982:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.298993:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.298996:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.299006:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299009:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.299024:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299028:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.299038:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299042:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.299051:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299054:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.299063:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299066:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.299075:0:324931:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.299080:0:324931:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431483.311964:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.311984:0:324934:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.311991:0:324934:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431483.312194:0:324934:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.312213:0:324934:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1789431483.312227:0:324934:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:2.0:1789431483.312799:0:324934:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.312807:0:324934:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.312811:0:324934:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e11c 00000080:00010000:2.0:1789431483.312819:0:324934:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e11c from it ffffbc42c174bcb8 00010000:00010000:2.0:1789431483.312823:0:324934:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:2.0:1789431483.312856:0:324934:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e11c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab730 expref: -99 pid: 323676 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:2.0:1789431483.314677:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314690:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.314702:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314706:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.314714:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314717:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.314725:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314728:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.314734:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314737:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.314743:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314746:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.314753:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314756:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.314763:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314766:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.314773:0:324934:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.314776:0:324934:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.341100:0:324936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.341118:0:324936:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.341126:0:324936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431483.341146:0:324936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.341159:0:324936:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431483.341205:0:324936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.341214:0:324936:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.341224:0:324936:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431483.341248:0:324936:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324936:1876350913458048:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431483.341325:0:324936:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1a7d900 going to sleep for 11 seconds 00000100:00100000:2.0:1789431483.347112:0:324936:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324936:1876350913458048:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431483.347139:0:324936:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347152:0:324936:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347165:0:324936:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347175:0:324936:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347188:0:324936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347196:0:324936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347205:0:324936:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431483.347216:0:324936:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4000 x1876350913458048/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431483.347230:0:324936:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347242:0:324936:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431483.347248:0:324936:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431483.347282:0:324936:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431483.347288:0:324936:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431483.347294:0:324936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.347297:0:324936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1c13cb8 00010000:00010000:2.0:1789431483.347300:0:324936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.347310:0:324936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431483.359528:0:324936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.359548:0:324936:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431483.359562:0:324936:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431483.359586:0:324936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431483.359919:0:324936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e1e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.359931:0:324936:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e1e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.359943:0:324936:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e1e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324936 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431483.359968:0:324936:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324936:1876350913458176:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431483.360101:0:324936:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1a7dcc0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431483.401485:0:324937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.401598:0:324937:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.401608:0:324937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431483.401640:0:324937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.401664:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431483.401684:0:324937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431483.401695:0:324937:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.401704:0:324937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.401708:0:324937:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431483.401717:0:324937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1c1bcb8 00010000:00010000:2.0:1789431483.401721:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431483.401731:0:324937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431483.401768:0:324937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431483.401778:0:324937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.401786:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431483.401795:0:324937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431483.401802:0:324937:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.401806:0:324937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431483.401809:0:324937:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431483.401813:0:324937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1c1bb58 00010000:00010000:2.0:1789431483.401817:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431483.401825:0:324937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431483.401993:0:324937:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.402011:0:324937:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=9 00000080:00200000:2.0:1789431483.402016:0:324937:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:2.0:1789431483.402032:0:324937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431483.402090:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324937 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.402099:0:324937:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324937 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.402110:0:324937:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324937 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.402141:0:324937:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324937:1876350913459328:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431483.402329:0:324937:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9900 going to sleep for 11 seconds 00000100:00100000:2.0:1789431483.405678:0:324937:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324937:1876350913459328:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431483.405712:0:324937:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.405732:0:324937:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.405741:0:324937:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.405761:0:324937:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.405776:0:324937:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406009:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406020:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406029:0:324937:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431483.406042:0:324937:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2101c00 x1876350913459328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431483.406058:0:324937:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406072:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431483.406087:0:324937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431483.406096:0:324937:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406104:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406112:0:324937:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e211 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795aba24 expref: -99 pid: 324937 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.406170:0:324937:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1789431483.406179:0:324937:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431483.406214:0:324937:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1789431483.406218:0:324937:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:2.0:1789431483.406225:0:324937:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431483.406245:0:324937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.406259:0:324937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1c1bb20 00010000:00010000:2.0:1789431483.406264:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431483.406273:0:324937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200008:2.0:1789431483.406329:0:324937:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.406337:0:324937:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431483.406363:0:324937:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.406373:0:324937:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431483.406382:0:324937:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431483.406389:0:324937:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.406394:0:324937:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431483.406398:0:324937:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431483.406402:0:324937:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1c1bcb8 00010000:00010000:2.0:1789431483.406405:0:324937:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431483.406412:0:324937:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431483.424439:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.424465:0:324940:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.424476:0:324940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:3.0:1789431483.424511:0:324940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.424546:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431483.424574:0:324940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431483.424585:0:324940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.424594:0:324940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431483.424598:0:324940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e218 00000080:00010000:3.0:1789431483.424608:0:324940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e218 from it ffffbc42c1c43cb8 00010000:00010000:3.0:1789431483.424612:0:324940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431483.424623:0:324940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431483.426756:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.426778:0:324940:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=17 00000080:00200000:3.0:1789431483.426785:0:324940:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=0 00000002:00010000:3.0:1789431483.426808:0:324940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431483.426902:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.426914:0:324940:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.426924:0:324940:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431483.426956:0:324940:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324940:1876350913459712:192.168.203.128@tcp:101:tail.0 00000100:00100000:3.0:1789431483.427073:0:324940:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1a7d600 going to sleep for 11 seconds 00000100:00100000:3.0:1789431483.430702:0:324940:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324940:1876350913459712:192.168.203.128@tcp:101:tail.0 00010000:00010000:3.0:1789431483.430735:0:324940:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430752:0:324940:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430760:0:324940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430768:0:324940:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430789:0:324940:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430801:0:324940:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431483.430807:0:324940:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430815:0:324940:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430824:0:324940:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430855:0:324940:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431483.430863:0:324940:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431483.430871:0:324940:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e21f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431483.430893:0:324940:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6a00 x1876350913459712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431483.431665:0:324940:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x68d:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431483.431704:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431483.431718:0:324940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431483.431728:0:324940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e218 from it ffffbc42c1c43970 00010000:00010000:3.0:1789431483.431733:0:324940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431483.431746:0:324940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431483.431829:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.431882:0:324940:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=17 00000080:00200000:3.0:1789431483.431888:0:324940:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=0 00000002:00010000:2.0:1789431483.434678:0:324940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431483.434764:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.434777:0:324940:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.434788:0:324940:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.434819:0:324940:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324940:1876350913459840:192.168.203.128@tcp:101:tail.0 00000100:00100000:2.0:1789431483.435227:0:324940:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431483.438344:0:324940:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324940:1876350913459840:192.168.203.128@tcp:101:tail.0 00010000:00010000:2.0:1789431483.438373:0:324940:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438391:0:324940:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438400:0:324940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438407:0:324940:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438417:0:324940:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438424:0:324940:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431483.438428:0:324940:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438435:0:324940:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438443:0:324940:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438452:0:324940:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431483.438458:0:324940:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.438467:0:324940:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e226 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431483.438490:0:324940:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2101c00 x1876350913459840/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431483.438508:0:324940:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x68d:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431483.438714:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431483.438731:0:324940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431483.441659:0:324940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e218 from it ffffbc42c1c43970 00010000:00010000:2.0:1789431483.441682:0:324940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431483.441700:0:324940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e218 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aba40 expref: -99 pid: 324907 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1789431483.442068:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.442082:0:324940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.442102:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.442106:0:324940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.442115:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.442118:0:324940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.442128:0:324940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.442142:0:324940:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=17 00000080:00200000:2.0:1789431483.442147:0:324940:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=0 00000002:00010000:2.0:1789431483.442179:0:324940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431483.442249:0:324940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e234 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.442258:0:324940:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e234 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.442269:0:324940:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e234 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324940 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.442306:0:324940:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324940:1876350913460096:192.168.203.128@tcp:101:tail.0 00000100:00100000:2.0:1789431483.442473:0:324940:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000080:00200000:2.0:1789431483.479322:0:324942:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.479350:0:324942:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=17 00000080:00200000:2.0:1789431483.479357:0:324942:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=0 00000002:00010000:2.0:1789431483.479390:0:324942:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431483.479469:0:324942:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e249 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.479482:0:324942:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e249 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324942 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.479494:0:324942:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e249 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324942 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.479663:0:324942:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324942:1876350913460864:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431483.480088:0:324942:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000080:00200000:3.0:1789431483.523017:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.523050:0:324945:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.526150:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526159:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.526175:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526179:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.526189:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526193:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.526201:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526205:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.526213:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526217:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.526228:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526232:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.526242:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526245:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.526254:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.526259:0:324944:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.527246:0:324944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.527273:0:324944:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=1 00000080:00200000:2.0:1789431483.527280:0:324944:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=getattr 00000002:00010000:2.0:1789431483.527316:0:324944:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431483.527400:0:324944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e273 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324944 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.527413:0:324944:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e273 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324944 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431483.527425:0:324944:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e273 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324944 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431483.527458:0:324944:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324944:1876350913462016:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431483.529686:0:324944:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0b8ec00 going to sleep for 11 seconds 00000080:00200000:2.0:1789431483.537348:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537362:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.537375:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537381:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.537391:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537394:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.537401:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537404:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.537411:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537414:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.537422:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537426:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.537435:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537439:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.537450:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537453:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.537462:0:324945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.537466:0:324945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.576047:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.576072:0:324952:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.576083:0:324952:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:3.0:1789431483.576112:0:324952:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.576135:0:324952:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431483.576153:0:324952:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431483.576162:0:324952:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.576170:0:324952:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431483.576174:0:324952:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:3.0:1789431483.576183:0:324952:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1c83cb8 00010000:00010000:3.0:1789431483.576187:0:324952:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431483.576204:0:324952:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431483.582120:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582128:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.582138:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582142:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.582150:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582153:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.582161:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582164:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.582170:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582173:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.582181:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582185:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.582192:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582194:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.582201:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582204:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.582211:0:324952:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.582214:0:324952:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.593276:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.593301:0:324954:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.593312:0:324954:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:3.0:1789431483.593340:0:324954:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.593363:0:324954:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431483.593386:0:324954:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431483.593395:0:324954:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.593404:0:324954:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431483.593408:0:324954:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:3.0:1789431483.593418:0:324954:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1c9bcb8 00010000:00010000:3.0:1789431483.593422:0:324954:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431483.593431:0:324954:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1789431483.598675:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598700:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.598715:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598720:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.598730:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598734:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.598742:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598745:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.598753:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598756:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.598764:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598768:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.598775:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598778:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.598787:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598791:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.598800:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.598804:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431483.599291:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599298:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.599308:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599311:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599322:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599325:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599335:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599338:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599348:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599351:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599360:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599363:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599371:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599375:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599384:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599387:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599394:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599399:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.599696:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599703:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.599713:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599716:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599725:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599728:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599736:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599739:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599747:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599750:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.599758:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599762:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599770:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599773:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599781:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599785:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.599795:0:324954:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.599800:0:324954:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431483.609066:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.609093:0:324957:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.609290:0:324957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:2.0:1789431483.609322:0:324957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.612029:0:324957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431483.612060:0:324957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1789431483.612074:0:324957:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.612085:0:324957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431483.612090:0:324957:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:2.0:1789431483.612102:0:324957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1cabcb8 00010000:00010000:2.0:1789431483.612106:0:324957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431483.612117:0:324957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431483.622856:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622869:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.622885:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622889:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.622900:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622903:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.622913:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622917:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.622926:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622929:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.622939:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622942:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.622950:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622953:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.622960:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622964:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.622971:0:324957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.622975:0:324957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.695181:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695203:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.695221:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695228:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.695241:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695247:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.695260:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695273:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.695291:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695298:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.695307:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695312:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.695323:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695326:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.695336:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695339:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.695347:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.695351:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.701856:0:324961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.701873:0:324961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, flags=1 00000080:00200008:2.0:1789431483.701890:0:324961:0:(llite_lib.c:2428:ll_setattr_raw()) START file pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d:[0x200000402:0x68e:0x0](ffff9811c4876c08) current size 4096, valid attrs 41, mode 416d, uid -1053097984, gid -26607, new size 0, atime 1789431483.000000000, mtime 1789431483.000000000, ctime 1789431483.000000000 00010000:00010000:2.0:1789431483.701998:0:324961:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e2b2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abb9e expref: -99 pid: 324951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431483.702060:0:324961:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e2b2 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abb9e expref: -99 pid: 324951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431483.702066:0:324961:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431483.702069:0:324961:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e2b2 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abb9e expref: -99 pid: 324951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1789431483.702091:0:324961:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431483.702103:0:324961:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101180 x1876350913463424/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431483.702126:0:324961:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324961:1876350913463424:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431483.702298:0:324961:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf8000 going to sleep for 11 seconds 00000080:00200000:3.0:1789431483.722617:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.722636:0:324964:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.722646:0:324964:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:3.0:1789431483.722667:0:324964:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.722686:0:324964:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431483.722701:0:324964:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1789431483.722715:0:324964:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.722722:0:324964:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431483.722725:0:324964:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:3.0:1789431483.722733:0:324964:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1cbbcb8 00010000:00010000:3.0:1789431483.722736:0:324964:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:3.0:1789431483.722746:0:324964:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1789431483.725317:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727659:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.727733:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727741:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.727758:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727762:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.727773:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727776:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.727786:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727788:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.727796:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727800:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.727813:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727816:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.727825:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727828:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.727853:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.727858:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431483.728257:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728264:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.728276:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728279:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728289:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728293:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728302:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728305:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728314:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728317:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728326:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728329:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728336:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728339:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728348:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728352:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728361:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728365:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.728563:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728569:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.728579:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728582:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728591:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728594:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728603:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728606:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728617:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728620:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.728630:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728633:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728640:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728643:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728652:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728655:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.728664:0:324964:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.728668:0:324964:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431483.746916:0:324967:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:2.0:1789431483.746952:0:324967:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.746977:0:324967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431483.746996:0:324967:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1789431483.747005:0:324967:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.747016:0:324967:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431483.747021:0:324967:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:2.0:1789431483.747030:0:324967:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1cc3cb8 00010000:00010000:2.0:1789431483.747034:0:324967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431483.747045:0:324967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1789431483.751166:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751176:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.751185:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751188:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.751194:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751197:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.751218:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751222:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.751230:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751233:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.751242:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751245:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.751252:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751256:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.751267:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751270:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.751279:0:324967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.751283:0:324967:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.800511:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:3.0:1789431483.800539:0:324972:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431483.800562:0:324972:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:3.0:1789431483.800593:0:324972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431483.800617:0:324972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431483.800637:0:324972:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1789431483.800647:0:324972:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431483.800657:0:324972:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431483.800661:0:324972:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:3.0:1789431483.800671:0:324972:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1cdbcb8 00010000:00010000:3.0:1789431483.800676:0:324972:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431483.800687:0:324972:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:3.0:1789431483.802231:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802248:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.802261:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802265:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.802273:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802276:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.802283:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802286:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.802293:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802296:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.802303:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802307:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.802314:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802316:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.802323:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802327:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.802334:0:324972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.802338:0:324972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.837114:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837122:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837135:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837139:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837149:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837152:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837162:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837165:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837173:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837176:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837186:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837189:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837198:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837201:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837211:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837215:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431483.837421:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837427:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.837437:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837440:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837450:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837454:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837462:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837465:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837473:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837476:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837484:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837487:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837497:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837499:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837507:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837510:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837519:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837522:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.837685:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837691:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.837701:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837704:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837713:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837715:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837723:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837727:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837736:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837738:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.837746:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837749:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837757:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837761:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837769:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837772:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.837780:0:324974:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.837784:0:324974:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431483.843985:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.844006:0:324977:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.844017:0:324977:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:2.0:1789431483.844043:0:324977:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.844065:0:324977:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431483.844083:0:324977:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1789431483.844092:0:324977:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.844101:0:324977:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431483.844105:0:324977:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:2.0:1789431483.844114:0:324977:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c154bcb8 00010000:00010000:2.0:1789431483.844118:0:324977:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431483.844134:0:324977:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1789431483.845735:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.845740:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.845748:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.845751:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.846123:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.846129:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.846302:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846308:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.846319:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846322:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.846330:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846333:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.846341:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846345:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.846352:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846356:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.846363:0:324977:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.846367:0:324977:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431483.853272:0:324976:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68e:0x0](ffff9811c4876c08), inode mode 41ed mask 3 00000080:00200000:0.0:1789431483.853278:0:324976:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_5b1611cbd47a55fb29e88f6539b686ae, dir=[0x200000402:0x68e:0x0](ffff9811c4876c08), target=test 00000020:00100000:0.0:1789431483.853354:0:324976:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431483.853362:0:324976:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6a00 x1876350913464320/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431483.853390:0:324976:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324976:1876350913464320:192.168.203.128@tcp:36:pjdfstest.65534 00000100:00100000:0.0:1789431483.853540:0:324976:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431483.867828:0:324976:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324976:1876350913464320:192.168.203.128@tcp:36:pjdfstest.65534 00000080:00200000:0.0:1789431483.868200:0:324976:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x690:0x0](ffff9811c48721c8) 00000080:00200000:0.0:1789431483.868205:0:324976:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x690:0x0], updating i_size 4 00000080:00200000:0.0:1789431483.898273:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898294:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431483.898305:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898310:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431483.898328:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898332:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431483.898340:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898343:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431483.898350:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898354:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431483.898361:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898365:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431483.898373:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898376:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431483.898392:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898395:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431483.898404:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.898408:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431483.899003:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.899009:0:324981:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, flags=16 00000080:00200000:0.0:1789431483.899016:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68e:0x0](ffff9811c4876c08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.899031:0:324981:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_5b1611cbd47a55fb29e88f6539b686ae, dir=[0x200000402:0x68e:0x0](ffff9811c4876c08), flags=0 00000080:00200000:0.0:1789431483.899035:0:324981:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_5b1611cbd47a55fb29e88f6539b686ae, dir=[0x200000402:0x68e:0x0](ffff9811c4876c08), intent=getattr 00000002:00010000:0.0:1789431483.899050:0:324981:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_5b1611cbd47a55fb29e88f6539b686ae,[0x0:0x0:0x0]) in obj [0x200000402:0x68e:0x0], intent: getattr flags 01 00010000:00010000:0.0:1789431483.899110:0:324981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68e:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324981 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.899119:0:324981:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324981 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.899130:0:324981:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324981 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431483.899162:0:324981:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324981:1876350913464448:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:0.0:1789431483.899291:0:324981:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431483.905126:0:324981:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324981:1876350913464448:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:0.0:1789431483.905163:0:324981:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905189:0:324981:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x690:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905207:0:324981:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905220:0:324981:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905233:0:324981:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905251:0:324981:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905260:0:324981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905267:0:324981:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431483.905279:0:324981:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd4a80 x1876350913464448/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431483.905292:0:324981:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905303:0:324981:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1680 (0 0) 00000080:00200000:0.0:1789431483.905310:0:324981:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68e:0x0] suppgids -1 -1: rc 0 00000080:00200000:0.0:1789431483.905342:0:324981:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x690:0x0], updating i_size 4 00000080:00200000:0.0:1789431483.905346:0:324981:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x690:0x0](ffff9811c48721c8): rc = 0 00000080:00010000:0.0:1789431483.905351:0:324981:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x690:0x0](ffff9811c48721c8) for lock 0x1ffb2ee1bd27e2d5 00000080:00010000:0.0:1789431483.905363:0:324981:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x690:0x0](ffff9811c48721c8) 00000080:00010000:0.0:1789431483.905367:0:324981:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e2d5 from it ffffbc42c1d33e00 00010000:00010000:0.0:1789431483.905370:0:324981:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.905582:0:324981:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431483.905622:0:324981:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68e:0x0](ffff9811c4876c08), inode mode 41ed mask 3 00000080:00200000:0.0:1789431483.905637:0:324981:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_5b1611cbd47a55fb29e88f6539b686ae, dir=[0x200000402:0x68e:0x0](ffff9811c4876c08) 00010000:00010000:0.0:1789431483.905673:0:324981:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431483.905728:0:324981:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431483.905735:0:324981:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431483.905738:0:324981:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2d5 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x690:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc62 expref: -99 pid: 324981 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1789431483.905766:0:324981:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431483.905786:0:324981:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4a80 x1876350913464704/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431483.905813:0:324981:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324981:1876350913464704:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431483.905948:0:324981:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431483.909215:0:324981:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324981:1876350913464704:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431483.909277:0:324981:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x690:0x0](ffff9811c48721c8) 00000080:00200000:2.0:1789431483.915870:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:2.0:1789431483.915893:0:324984:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431483.915904:0:324984:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:2.0:1789431483.915933:0:324984:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431483.915966:0:324984:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1789431483.915985:0:324984:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1789431483.915994:0:324984:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431483.916004:0:324984:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:2.0:1789431483.916008:0:324984:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e242 00000080:00010000:2.0:1789431483.916017:0:324984:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e242 from it ffffbc42c1d33cb8 00010000:00010000:2.0:1789431483.916022:0:324984:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1789431483.916038:0:324984:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:3.0:1789431483.933452:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.933478:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.933497:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.933503:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.933819:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.933825:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.934006:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934011:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.934023:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934027:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.934037:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934041:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.934050:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934053:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.934065:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934068:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.934078:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.934082:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431483.936332:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936351:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.936365:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936369:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.936377:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936380:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.936594:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936599:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.936608:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936611:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.936618:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936621:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.936628:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936631:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.936640:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936643:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.936650:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.936654:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431483.937720:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937817:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431483.937886:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937891:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.937904:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937908:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.937919:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937922:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.937931:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937934:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431483.937943:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937947:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.937956:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937959:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.937969:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937973:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431483.937983:0:324984:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:3.0:1789431483.937987:0:324984:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431483.959678:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959688:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431483.959699:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959702:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.959711:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959714:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.959722:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959726:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.959735:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959739:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431483.959748:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959752:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.959760:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959763:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.959769:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959772:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431483.959781:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.959784:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431483.960230:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:2.0:1789431483.960253:0:324986:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, flags=0 00000080:00200000:2.0:1789431483.960263:0:324986:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 3 00000080:00200000:2.0:1789431483.960270:0:324986:0:(namei.c:2254:ll_rmdir()) VFS Op:name=pjdfstest_f5280a3d1296255edcf0a80d54cdaa9d, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08) 00010000:00010000:2.0:1789431483.960318:0:324986:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1789431483.960390:0:324986:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e2ce lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abc38 expref: -99 pid: 324976 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431483.960442:0:324986:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e2ce lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc38 expref: -99 pid: 324976 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431483.960452:0:324986:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:2.0:1789431483.960458:0:324986:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1789431483.960462:0:324986:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e2ce lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68e:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc38 expref: -99 pid: 324976 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431483.960479:0:324986:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031800/0x1ffb2ee1bd27e242 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abacc expref: -99 pid: 324942 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000020:00100000:2.0:1789431483.960499:0:324986:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431483.960514:0:324986:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101180 x1876350913464960/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431483.960542:0:324986:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324986:1876350913464960:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431483.960668:0:324986:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf80c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431483.964825:0:324986:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324986:1876350913464960:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431483.965036:0:324986:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x68e:0x0](ffff9811c4876c08) 00000080:00200000:0.0:1789431483.978753:0:324989:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200008:0.0:1789431483.978782:0:324989:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_0b1588e741598d5a8daa115ffde33a93:[0x200000402:0x68d:0x0](ffff9811c4872a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431483.978799:0:324989:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08),name=pjdfstest_0b1588e741598d5a8daa115ffde33a93 00000002:00010000:0.0:1789431483.978827:0:324989:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x68d:0x0]) in obj [0x200000402:0x68d:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431483.978934:0:324989:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.978947:0:324989:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.978960:0:324989:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431483.978995:0:324989:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324989:1876350913465088:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431483.979142:0:324989:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431483.984527:0:324989:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324989:1876350913465088:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431483.984573:0:324989:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984589:0:324989:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 4/1,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984604:0:324989:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984617:0:324989:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984632:0:324989:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984647:0:324989:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984656:0:324989:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431483.984669:0:324989:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7b80 x1876350913465088/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431499 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431483.984684:0:324989:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.984700:0:324989:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1677 (0 0) 00000080:00200000:0.0:1789431483.984708:0:324989:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x68d:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431483.984811:0:324989:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x68d:0x0], updating i_size 4096 00000080:00010000:0.0:1789431483.984820:0:324989:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e2dc 00000080:00010000:0.0:1789431483.984826:0:324989:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:0.0:1789431483.985108:0:324989:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e2dc from it ffffbc42c1d5bcb8 00010000:00010000:0.0:1789431483.985114:0:324989:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.985125:0:324989:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431483.993545:0:324989:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08), inode mode 41ed mask 1 00000080:00200000:0.0:1789431483.993586:0:324989:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), flags=17 00000080:00200000:0.0:1789431483.993593:0:324989:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x68d:0x0](ffff9811c4872a08), intent=0 00000002:00010000:0.0:1789431483.993628:0:324989:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x68d:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431483.993709:0:324989:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e2e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.993722:0:324989:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e2e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431483.993733:0:324989:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e2e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x68d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 324989 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431483.993766:0:324989:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324989:1876350913465216:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431483.993915:0:324989:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00010000:00010000:0.0:1789431484.047694:0:324989:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431484.047712:0:324989:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431484.047723:0:324989:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e2dc from it ffffbc42c1d5b9a0 00010000:00010000:0.0:1789431484.047728:0:324989:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431484.047736:0:324989:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200008:2.0:1789431484.053924:0:324907:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431484.053940:0:324907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431484.053971:0:324907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431484.053993:0:324907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431484.054012:0:324907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431484.054020:0:324907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431484.054030:0:324907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431484.054034:0:324907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431484.054044:0:324907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431484.054048:0:324907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431484.054059:0:324907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00000080:00200000:2.0:1789431484.054092:0:324907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431484.054099:0:324907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431484.054106:0:324907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1789431484.054115:0:324907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00000080:00200000:2.0:1789431484.054122:0:324907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431484.054126:0:324907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431484.054128:0:324907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431484.054134:0:324907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1de3b58 00010000:00010000:2.0:1789431484.054137:0:324907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1789431484.054145:0:324907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431484.054154:0:324907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.054162:0:324907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:2.0:1789431484.054171:0:324907:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431484.054175:0:324907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431484.054189:0:324907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431484.054197:0:324907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431484.054206:0:324907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431484.054214:0:324907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431484.054218:0:324907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431484.054220:0:324907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431484.054224:0:324907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1de3cb8 00010000:00010000:2.0:1789431484.054226:0:324907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431484.054234:0:324907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00000080:00200000:2.0:1789431484.054257:0:324907:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431484.054264:0:324907:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431484.054271:0:324907:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431484.054278:0:324907:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431484.054284:0:324907:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431484.054288:0:324907:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431484.054291:0:324907:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431484.054295:0:324907:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1de3c18 00010000:00010000:2.0:1789431484.054298:0:324907:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431484.054306:0:324907:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1789431484.054314:0:324907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.054319:0:324907:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:2.0:1789431484.054324:0:324907:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:2.0:1789431484.065059:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431484.065076:0:324992:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431484.065085:0:324992:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431484.065121:0:324992:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431484.065146:0:324992:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1789431484.065168:0:324992:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00000080:00200000:2.0:1789431484.065177:0:324992:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431484.065188:0:324992:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431484.065192:0:324992:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e1e0 00000080:00010000:2.0:1789431484.065200:0:324992:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e1e0 from it ffffbc42c1d33cb8 00010000:00010000:2.0:1789431484.065204:0:324992:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 0 00010000:00010000:2.0:1789431484.065214:0:324992:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 0 00000002:00010000:3.0:1789431484.066636:0:324991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_0b1588e741598d5a8daa115ffde33a93,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431484.066721:0:324991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431484.066734:0:324991:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431484.066746:0:324991:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324991 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431484.066778:0:324991:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324991:1876350913466368:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431484.067052:0:324991:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1a7dc00 going to sleep for 11 seconds 00000080:00200000:2.0:1789431484.070258:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070273:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431484.070286:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070291:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431484.070300:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070304:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431484.070312:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070315:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431484.070324:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070327:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431484.070335:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070342:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431484.070350:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070355:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431484.070367:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070374:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431484.070386:0:324992:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431484.070391:0:324992:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00010000:00010000:0.0:1789431484.070436:0:6160:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1789431484.070446:0:6160:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070454:0:6160:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.070460:0:6160:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070469:0:6160:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070478:0:6160:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 4/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070496:0:6160:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431484.070540:0:6160:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 3/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070558:0:6160:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431484.070587:0:6160:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e30d lrc: 2/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abd34 expref: -99 pid: 324991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.070596:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431484.070599:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431484.070601:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431484.070620:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913466624:192.168.203.128@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1789431484.071524:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071539:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071561:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071569:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071577:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 11 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071583:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.071587:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071593:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071599:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071606:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.071612:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071619:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e314 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 11 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071646:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071663:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071670:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071676:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071682:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 10 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071688:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.071691:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071696:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071702:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071708:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.071711:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071717:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e31b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 10 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071730:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071737:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071743:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071749:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071755:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 9 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071761:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.071764:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071770:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071775:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071781:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.071783:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071789:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e322 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 9 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071802:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071809:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071816:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071822:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071827:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 8 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071853:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.071856:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071862:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071868:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071874:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.071877:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071882:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e329 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 8 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071898:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071904:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071911:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071917:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071924:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 7 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071930:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.071933:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071939:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071946:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071952:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.071955:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071962:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e330 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 7 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071976:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071983:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071989:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.071994:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072000:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 6 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072005:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ee00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.072008:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072013:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072019:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072024:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.072028:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072034:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e337 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 6 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072046:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072053:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072059:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072065:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072071:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 5 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072076:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.072079:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072085:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072091:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072097:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.072099:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072105:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e33e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 5 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072116:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072123:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072129:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072285:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072292:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072300:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.072305:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072319:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072333:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072339:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.072343:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.072353:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e345 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431484.072380:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913466496:192.168.203.128@tcp:63:ll_sa.0 00000080:00200000:0.0:1789431484.072453:0:324991:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x68d:0x0], updating i_size 4096 00000080:00200000:0.0:1789431484.072457:0:324991:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x68d:0x0](ffff9811c4872a08): rc = 0 00000080:00010000:0.0:1789431484.072466:0:324991:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e2dc 00000080:00010000:0.0:1789431484.072490:0:324991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x68d:0x0](ffff9811c4872a08) 00000080:00010000:0.0:1789431484.072495:0:324991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e2dc from it ffffbc42c1d23e08 00010000:00010000:0.0:1789431484.072499:0:324991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431484.072509:0:324991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431484.072545:0:324991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431484.072565:0:324991:0:(namei.c:2254:ll_rmdir()) VFS Op:name=pjdfstest_0b1588e741598d5a8daa115ffde33a93, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:0.0:1789431484.072604:0:324991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 0 00010000:00010000:0.0:1789431484.072624:0:324991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431484.072852:0:324991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431484.072866:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431484.073008:0:324991:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 1/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431484.073017:0:324991:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 0 00010000:00010000:0.0:1789431484.073023:0:324991:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1789431484.073026:0:324991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e2dc lrc: 0/0,0 mode: --/PR res: [0x200000402:0x68d:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abc9a expref: -99 pid: 324989 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431484.073056:0:324991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e1e0 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab98a expref: -99 pid: 324936 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 0 00000020:00100000:0.0:1789431484.073072:0:324991:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431484.073081:0:324991:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7100 x1876350913466880/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431484.073109:0:324991:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324991:1876350913466880:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431484.073203:0:324991:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431484.076522:0:324991:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324991:1876350913466880:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431484.076593:0:324991:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x68d:0x0](ffff9811c4872a08) 00000100:00100000:0.0:1789431484.137055:0:324996:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c85256c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431484.141387:0:324996:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324996:1876350913467904:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431484.141419:0:324996:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141436:0:324996:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141444:0:324996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141452:0:324996:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141463:0:324996:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141471:0:324996:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ec00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.141477:0:324996:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141485:0:324996:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141493:0:324996:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141500:0:324996:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.141507:0:324996:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.141525:0:324996:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e36f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431484.141559:0:324996:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913467904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431500 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431484.141576:0:324996:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431484.141613:0:324996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431484.141628:0:324996:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:0.0:1789431484.141638:0:324996:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1d6b9a0 00010000:00010000:0.0:1789431484.141642:0:324996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431484.141669:0:324996:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431484.142207:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142424:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431484.142444:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142449:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431484.142459:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142462:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431484.142474:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142477:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431484.142485:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142488:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431484.142499:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142606:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431484.142626:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142631:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431484.142641:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142644:0:324996:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431484.142651:0:324996:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431484.142666:0:324996:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0dcd980, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431484.142672:0:324996:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431484.142703:0:324996:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431484.142772:0:324996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.142783:0:324996:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431484.142797:0:324996:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431484.142807:0:324996:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5f80 x1876350913468160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431484.142817:0:324996:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431484.142876:0:324996:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324996:1876350913468160:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431484.143555:0:324996:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431484.148506:0:324996:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:324996:1876350913468160:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431484.148617:0:324996:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148634:0:324996:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148642:0:324996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148720:0:324996:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148731:0:324996:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148739:0:324996:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431484.148744:0:324996:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148756:0:324996:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148764:0:324996:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148772:0:324996:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431484.148780:0:324996:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431484.148787:0:324996:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e376 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324996 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431484.148811:0:324996:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5f80 x1876350913468160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431500 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431484.148828:0:324996:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431484.149632:0:324996:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431484.149773:0:324996:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431484.149786:0:324996:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1d6b9a0 00010000:00010000:0.0:1789431484.149792:0:324996:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431484.149801:0:324996:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431485.094005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431485.715183:0:325128:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:0.0:1789431487.143012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431488.166295:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431490.215955:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431490.347516:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913468672:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431490.347605:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913468928:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431490.347637:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789431490.347644:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913468800:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431490.355077:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9811d1e45800: committing for last_committed 4294973991 gen 1 00000100:00100000:3.0:1789431490.355118:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913468800:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431490.355133:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913468672:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431490.355147:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc5500 x1876350913468928/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431506 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431490.355159:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913468928:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431491.238361:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431493.287955:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431493.359779:0:325844:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_07: symlink returns ELOOP if too many symbolic links were encountered in translating the name2 path name ========================================================== 20:18:12 (1789431492) 00010000:00010000:2.0:1789431494.310345:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431495.462989:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913469056:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431495.463074:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913469312:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431495.463109:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431495.463115:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913469184:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431495.465980:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913469056:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431495.467296:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913469184:192.168.203.128@tcp:400:kworker.0 00010000:00080000:2.0:1789431495.467317:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc4a80 x1876350913469312/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431511 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431495.467329:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913469312:192.168.203.128@tcp:400:kworker.0 00010000:00010000:0.0:1789431496.361706:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431497.383909:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431497.653509:0:326174:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/07.t against lustre filesystem 00000080:00200000:3.0:1789431497.968070:0:326186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431497.968099:0:326186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431497.968404:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1789431497.968422:0:326186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:3.0:1789431497.968431:0:326186:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431497.968437:0:326186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431497.968441:0:326186:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:3.0:1789431497.968449:0:326186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1dbbb58 00010000:00010000:3.0:1789431497.968462:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:3.0:1789431497.968472:0:326186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1789431497.968489:0:326186:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431497.968565:0:326186:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=13 00000080:00200000:3.0:1789431497.968569:0:326186:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:3.0:1789431497.968584:0:326186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431497.968690:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 326186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431497.968702:0:326186:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431497.968714:0:326186:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326186 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431497.968737:0:326186:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326186:1876350913469440:192.168.203.128@tcp:101:lfs.0 00000100:00100000:3.0:1789431497.968829:0:326186:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:2.0:1789431497.983564:0:326186:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326186:1876350913469440:192.168.203.128@tcp:101:lfs.0 00010000:00010000:2.0:1789431497.983591:0:326186:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983608:0:326186:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983615:0:326186:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983630:0:326186:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983640:0:326186:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983652:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983659:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983666:0:326186:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431497.983677:0:326186:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4700 x1876350913469440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431513 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431497.983691:0:326186:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983702:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431497.983712:0:326186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431497.983720:0:326186:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983727:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983733:0:326186:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e37d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795abe06 expref: -99 pid: 326186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431497.983757:0:326186:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1789431497.983763:0:326186:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431497.983801:0:326186:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1789431497.983805:0:326186:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:2.0:1789431497.983811:0:326186:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:2.0:1789431497.983824:0:326186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431497.983827:0:326186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1dbbb20 00010000:00010000:2.0:1789431497.983874:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431497.983882:0:326186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200008:2.0:1789431497.983924:0:326186:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1789431497.983930:0:326186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431497.983950:0:326186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431497.983958:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431497.983965:0:326186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431497.983972:0:326186:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431497.983976:0:326186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431497.983979:0:326186:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:2.0:1789431497.983982:0:326186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1dbbcb8 00010000:00010000:2.0:1789431497.983984:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431497.983990:0:326186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200008:2.0:1789431497.984547:0:326186:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1789431497.984552:0:326186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431497.984563:0:326186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431497.984572:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431497.984582:0:326186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1789431497.984588:0:326186:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431497.984592:0:326186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431497.984595:0:326186:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431497.984599:0:326186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1dbbcb8 00010000:00010000:2.0:1789431497.984602:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431497.984616:0:326186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1789431497.984730:0:326186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431497.984739:0:326186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431497.984745:0:326186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1789431497.984753:0:326186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1789431497.984759:0:326186:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431497.984763:0:326186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431497.984765:0:326186:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e154 00000080:00010000:2.0:1789431497.984768:0:326186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e154 from it ffffbc42c1dbbbb0 00010000:00010000:2.0:1789431497.984771:0:326186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:2.0:1789431497.984777:0:326186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1789431497.984785:0:326186:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:2.0:1789431497.984791:0:326186:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:2.0:1789431497.984806:0:326186:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffd3f060c30 00000080:00200000:2.0:1789431497.984817:0:326186:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:2.0:1789431497.984881:0:326186:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1789431497.984923:0:326186:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1789431497.984930:0:326186:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431497.984933:0:326186:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0031600/0x1ffb2ee1bd27e154 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ab7d8 expref: -99 pid: 324392 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000020:00100000:2.0:1789431497.984955:0:326186:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431497.984965:0:326186:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102300 x1876350913469696/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431497.984986:0:326186:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326186:1876350913469696:192.168.203.128@tcp:36:lfs.0 00000100:00100000:2.0:1789431497.985168:0:326186:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431497.985912:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913469824:192.168.203.128@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1789431497.990500:0:326186:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326186:1876350913469696:192.168.203.128@tcp:36:lfs.0 00000080:00200008:3.0:1789431497.990597:0:326186:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200000:3.0:1789431498.163094:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163220:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431498.163239:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163243:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431498.163254:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163258:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431498.163267:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163270:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431498.163279:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163283:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431498.163293:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163297:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431498.163306:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163309:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431498.163317:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163320:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431498.163328:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431498.163339:0:326189:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf1800, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431498.163346:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431498.163371:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431498.163548:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e38b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431498.163559:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e38b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431498.163571:0:326189:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431498.163583:0:326189:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6680 x1876350913470208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431498.163592:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e38b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431498.163614:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913470208:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431498.164138:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34e40 going to sleep for 11 seconds 00000080:00200000:2.0:1789431498.191038:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.191045:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.191059:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.191062:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.191071:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.191086:0:326189:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0e900, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431498.191092:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431498.191114:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431498.191165:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.191174:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431498.191185:0:326189:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431498.191195:0:326189:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913471232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431498.191204:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.191224:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471232:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.191414:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.194643:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471232:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.194668:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194682:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194824:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194868:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194878:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194884:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.194888:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194894:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194905:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194911:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.194917:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.194922:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3a7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.194941:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913471232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.194955:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.195100:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431498.195115:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00010000:2.0:1789431498.195124:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db39a0 00010000:00010000:2.0:1789431498.195128:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:2.0:1789431498.195137:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:2.0:1789431498.195568:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195578:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431498.195590:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195594:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.195602:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195605:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.195613:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195616:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.195625:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195628:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.195636:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195639:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.195648:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195651:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.195657:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195660:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.195667:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.195677:0:326189:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0f500, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431498.195683:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431498.195827:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libutil.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431498.195902:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.195910:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431498.195920:0:326189:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431498.195928:0:326189:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913471488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431498.195936:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.195959:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471488:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.196425:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.202904:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471488:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.202928:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202940:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202946:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202953:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202962:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202968:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.202972:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202977:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202983:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.202998:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.203004:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.203010:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3ae lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.203026:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913471488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.203038:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.203065:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431498.203078:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00010000:2.0:1789431498.203086:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db39a0 00010000:00010000:2.0:1789431498.203090:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:2.0:1789431498.203098:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:2.0:1789431498.203478:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203489:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431498.203502:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203506:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.203513:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203516:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.203523:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203526:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.203535:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203538:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.203546:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203550:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.203558:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203561:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.203567:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203570:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.203576:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.203580:0:326189:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00010000:00010000:3.0:1789431498.413550:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789431498.547073:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.547108:0:326189:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431498.547115:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431498.547169:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .proverc,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431498.547251:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.547264:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.547275:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.547306:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471872:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.547731:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.551982:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913471872:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.552014:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552030:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552039:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552046:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552057:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552064:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.552069:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552077:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552086:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552094:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.552100:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.552106:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3bc lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.552126:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913471872/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.552143:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.552177:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1789431498.552192:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00010000:2.0:1789431498.552202:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db3930 00010000:00010000:2.0:1789431498.552206:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1789431498.552215:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1789431498.554533:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.554572:0:326189:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431498.554580:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431498.554640:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .prove,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431498.554718:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.554731:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.554742:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.554773:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472000:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.554940:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.559125:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472000:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.559156:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559173:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559180:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559187:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559196:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559202:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.559206:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559213:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559221:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559228:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.559234:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.559240:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3c3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.559262:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913472000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.559285:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.559323:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431498.559337:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00010000:2.0:1789431498.559361:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db3930 00010000:00010000:2.0:1789431498.559365:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1789431498.559374:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1789431498.631765:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.631784:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.633462:0:326189:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431498.633477:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431498.633540:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: testrules.yml,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431498.633625:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.633638:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.633649:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.633680:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472128:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.633901:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.638063:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472128:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.638096:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638115:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638124:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638132:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638143:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638150:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.638155:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638163:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638171:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638179:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.638186:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638198:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e3ca lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.638220:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913472128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.638238:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.638275:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1789431498.638291:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00010000:2.0:1789431498.638302:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db3930 00010000:00010000:2.0:1789431498.638306:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1789431498.638315:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1789431498.638403:0:326189:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.638424:0:326189:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431498.638429:0:326189:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431498.638462:0:326189:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: t,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431498.638513:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638521:0:326189:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.638532:0:326189:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431498.638557:0:326189:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472256:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431498.638666:0:326189:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431498.644182:0:326189:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326189:1876350913472256:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431498.644216:0:326189:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644234:0:326189:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644243:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644251:0:326189:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644262:0:326189:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644269:0:326189:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431498.644274:0:326189:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644283:0:326189:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644292:0:326189:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644301:0:326189:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431498.644308:0:326189:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431498.644315:0:326189:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e3d1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326189 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431498.644338:0:326189:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913472256/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431498.644356:0:326189:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431498.644393:0:326189:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1789431498.644409:0:326189:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00010000:2.0:1789431498.644422:0:326189:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1db38d0 00010000:00010000:2.0:1789431498.644426:0:326189:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1789431498.644436:0:326189:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1789431498.906359:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431498.906402:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.906427:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431498.906455:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431498.906464:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.906475:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431498.906479:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:0.0:1789431498.906489:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1d9bb58 00010000:00010000:0.0:1789431498.906493:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431498.906503:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431498.906521:0:326190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.906532:0:326190:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431498.906543:0:326190:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.906549:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.906564:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.906573:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1789431498.906582:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1789431498.906588:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.906593:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.906595:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.906599:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1d9bcb8 00010000:00010000:0.0:1789431498.906602:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1789431498.906609:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1789431498.906624:0:326190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431498.906629:0:326190:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.906633:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.906640:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.906646:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1789431498.906654:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1789431498.906661:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.906665:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.906667:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.906671:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1d9bcb8 00010000:00010000:0.0:1789431498.906674:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1789431498.906681:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200008:0.0:1789431498.906705:0:326190:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.906709:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431498.906714:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.906720:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431498.906728:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431498.906735:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.906739:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431498.906741:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:0.0:1789431498.906744:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1d9bcb8 00010000:00010000:0.0:1789431498.906748:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431498.906761:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431498.906775:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431498.906780:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.906786:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431498.906794:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431498.906800:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.906804:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431498.906806:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:0.0:1789431498.906809:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1d9bb58 00010000:00010000:0.0:1789431498.906812:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431498.906820:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431498.906827:0:326190:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.907275:0:326190:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431498.907284:0:326190:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.907288:0:326190:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.907300:0:326190:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.907309:0:326190:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1789431498.907318:0:326190:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1789431498.907325:0:326190:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.907328:0:326190:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.907331:0:326190:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.907334:0:326190:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1d9bcb8 00010000:00010000:0.0:1789431498.907337:0:326190:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1789431498.907344:0:326190:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1789431498.933694:0:326194:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431498.933724:0:326194:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.933734:0:326194:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.933774:0:326194:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.933991:0:326194:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1789431498.934012:0:326194:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1789431498.934021:0:326194:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.934032:0:326194:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.934037:0:326194:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.934047:0:326194:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c154bcb8 00010000:00010000:0.0:1789431498.934052:0:326194:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1789431498.934062:0:326194:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1789431498.939533:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939556:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431498.939571:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939576:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.939587:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939592:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.939669:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939674:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.939688:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939692:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.939703:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939708:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.939718:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939721:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.939730:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939733:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.939741:0:326195:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.939745:0:326195:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431498.949240:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431498.949268:0:326197:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431498.949276:0:326197:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431498.949312:0:326197:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431498.949334:0:326197:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1789431498.949350:0:326197:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1789431498.949359:0:326197:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431498.949369:0:326197:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431498.949373:0:326197:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:3.0:1789431498.949382:0:326197:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c154bcb8 00010000:00010000:3.0:1789431498.949386:0:326197:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:3.0:1789431498.949395:0:326197:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1789431498.950394:0:326198:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1789431498.950407:0:326198:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431498.950414:0:326198:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431498.950417:0:326198:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:1.0:1789431498.950422:0:326198:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1563cb8 00010000:00010000:1.0:1789431498.950426:0:326198:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1789431498.950434:0:326198:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1789431498.952406:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431498.952434:0:326199:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.952443:0:326199:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.953158:0:326199:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.953183:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1789431498.953201:0:326199:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1789431498.953209:0:326199:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.953218:0:326199:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.953222:0:326199:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.953230:0:326199:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1577cb8 00010000:00010000:0.0:1789431498.953234:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1789431498.953244:0:326199:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1789431498.959540:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959566:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431498.959586:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959591:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.959601:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959604:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.959627:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959630:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.959641:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959644:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.959654:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959658:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.959668:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959672:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.959682:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959685:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.959695:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.959707:0:326199:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e00c1200, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431498.959716:0:326199:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431498.959751:0:326199:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libsigsegv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431498.960121:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431498.960135:0:326199:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431498.960230:0:326199:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431498.960246:0:326199:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5f80 x1876350913472768/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431498.960257:0:326199:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431498.960288:0:326199:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913472768:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431498.960762:0:326199:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:1.0:1789431498.970019:0:326199:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913472768:192.168.203.128@tcp:101:awk.0 00010000:00010000:1.0:1789431498.970052:0:326199:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970069:0:326199:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970078:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970086:0:326199:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970097:0:326199:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970104:0:326199:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431498.970109:0:326199:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970118:0:326199:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970126:0:326199:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970134:0:326199:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431498.970215:0:326199:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.970223:0:326199:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e3df lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431498.970246:0:326199:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5f80 x1876350913472768/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431498.970264:0:326199:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431498.970301:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1789431498.970315:0:326199:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00010000:1.0:1789431498.970324:0:326199:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c15779a0 00010000:00010000:1.0:1789431498.970329:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1789431498.970336:0:326199:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1789431498.972266:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972286:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431498.972296:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972300:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.972308:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972310:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.972318:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972320:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.972327:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972330:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431498.972336:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972340:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.972346:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972349:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.972356:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972359:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431498.972366:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.972370:0:326197:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431498.973259:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431498.973272:0:326197:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.973280:0:326197:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431498.973311:0:326197:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.973334:0:326197:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431498.973353:0:326197:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1789431498.973362:0:326197:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.973370:0:326197:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431498.973374:0:326197:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431498.973382:0:326197:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c154bcb8 00010000:00010000:0.0:1789431498.973385:0:326197:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431498.973394:0:326197:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1789431498.973423:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.973427:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:0.0:1789431498.973433:0:326197:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:0.0:1789431498.973471:0:326197:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:0.0:1789431498.973485:0:326197:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:0.0:1789431498.974338:0:326197:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431498.974346:0:326197:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431498.974362:0:326197:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431498.974375:0:326197:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431498.974394:0:326197:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431498.974401:0:326197:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431498.974407:0:326197:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431498.974410:0:326197:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:0.0:1789431498.974417:0:326197:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c154bcb8 00010000:00010000:0.0:1789431498.974420:0:326197:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431498.974428:0:326197:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431498.975699:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975717:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431498.975731:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975735:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.975742:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975745:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.975751:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975754:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.975760:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975763:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.975770:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975773:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.975782:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975910:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.975927:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975931:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.975941:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.975955:0:326199:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c55ad680, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431498.975963:0:326199:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431498.975992:0:326199:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libreadline.so.7,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431498.976533:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.976546:0:326199:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431498.976562:0:326199:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431498.976578:0:326199:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913473024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431498.976590:0:326199:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431498.976618:0:326199:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473024:192.168.203.128@tcp:101:awk.0 00000100:00100000:1.0:1789431498.979266:0:326199:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc2180 going to sleep for 11 seconds 00000080:00200000:2.0:1789431498.980883:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980905:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431498.980918:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980923:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.980933:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980937:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.980945:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980949:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.980956:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980959:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431498.980966:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980970:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.980979:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980982:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.980991:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.980994:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431498.981002:0:326198:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431498.981006:0:326198:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:1.0:1789431498.988928:0:326199:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473024:192.168.203.128@tcp:101:awk.0 00010000:00010000:1.0:1789431498.988958:0:326199:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.988974:0:326199:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.988983:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.988992:0:326199:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989002:0:326199:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989008:0:326199:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ec00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431498.989014:0:326199:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989021:0:326199:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989030:0:326199:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989037:0:326199:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431498.989044:0:326199:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989050:0:326199:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3e6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431498.989072:0:326199:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913473024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431498.989086:0:326199:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431498.989119:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1789431498.989133:0:326199:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00010000:1.0:1789431498.989154:0:326199:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c15779a0 00010000:00010000:1.0:1789431498.989158:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:1.0:1789431498.989167:0:326199:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:1.0:1789431498.989612:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989624:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431498.989633:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989638:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.989644:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989648:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.989655:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989657:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.989665:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989667:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.989673:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989677:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.989684:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989687:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.989693:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989696:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.989702:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.989714:0:326199:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf6900, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431498.989721:0:326199:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431498.989742:0:326199:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libmpfr.so.4,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431498.989799:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.989807:0:326199:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431498.989819:0:326199:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431498.989827:0:326199:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913473280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431498.989867:0:326199:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431498.989891:0:326199:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473280:192.168.203.128@tcp:101:awk.0 00000100:00100000:1.0:1789431498.990103:0:326199:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc2180 going to sleep for 11 seconds 00000080:00200000:0.0:1789431498.990523:0:326197:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431498.990534:0:326197:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:0.0:1789431498.990539:0:326197:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3857, max_age 3872 00800000:00000004:0.0:1789431498.990549:0:326197:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3857, max_age 3872 00000100:00100000:0.0:1789431498.990573:0:326197:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326197:1876350913473408:192.168.203.128@tcp:41:df.0 00000100:00100000:0.0:1789431498.990606:0:326197:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431498.995525:0:326197:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326197:1876350913473408:192.168.203.128@tcp:41:df.0 00800000:00000004:0.0:1789431498.995563:0:326197:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:0.0:1789431498.995572:0:326197:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:0.0:1789431498.995578:0:326197:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000100:00100000:1.0:1789431498.997715:0:326199:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473280:192.168.203.128@tcp:101:awk.0 00010000:00010000:1.0:1789431498.997742:0:326199:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997758:0:326199:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997766:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997773:0:326199:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997797:0:326199:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997804:0:326199:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431498.997808:0:326199:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997816:0:326199:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997824:0:326199:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997853:0:326199:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431498.997861:0:326199:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431498.997868:0:326199:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e3ed lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431498.997892:0:326199:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913473280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431498.997909:0:326199:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431498.997945:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1789431498.997959:0:326199:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00010000:1.0:1789431498.997969:0:326199:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c15779a0 00010000:00010000:1.0:1789431498.997973:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1789431498.997982:0:326199:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1789431498.999584:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999606:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431498.999620:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999625:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.999631:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999634:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.999641:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999643:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.999650:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999653:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431498.999660:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999663:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.999673:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999675:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.999683:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999685:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431498.999691:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431498.999706:0:326199:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf7200, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431498.999714:0:326199:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431498.999746:0:326199:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libgmp.so.10,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.000484:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.000499:0:326199:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.000517:0:326199:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.000533:0:326199:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913473664/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.000543:0:326199:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.000570:0:326199:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473664:192.168.203.128@tcp:101:awk.0 00000100:00100000:1.0:1789431499.001427:0:326199:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc2180 going to sleep for 10 seconds 00000100:00100000:1.0:1789431499.005829:0:326199:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326199:1876350913473664:192.168.203.128@tcp:101:awk.0 00010000:00010000:1.0:1789431499.005940:0:326199:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.005955:0:326199:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.005967:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.005975:0:326199:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.005985:0:326199:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.005992:0:326199:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ec00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.005998:0:326199:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.006007:0:326199:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.006016:0:326199:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.006024:0:326199:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.006031:0:326199:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.006039:0:326199:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3f4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326199 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.006060:0:326199:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913473664/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431514 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.006078:0:326199:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.006117:0:326199:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1789431499.006133:0:326199:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00010000:1.0:1789431499.006668:0:326199:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c15779a0 00010000:00010000:1.0:1789431499.006675:0:326199:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1789431499.006685:0:326199:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:1.0:1789431499.007307:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007323:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.007336:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007340:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.007349:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007352:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.007361:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007364:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.007371:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007374:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.007383:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007387:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.007396:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007399:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.007977:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.007986:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008005:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008009:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431499.008211:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008216:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.008227:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008231:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008240:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008243:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008252:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008255:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008262:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008266:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008273:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008276:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008284:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008287:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008295:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008298:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008306:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008309:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431499.008466:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008472:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.008482:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008485:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008493:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008497:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008506:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008509:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008517:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008521:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008529:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008532:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008541:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008545:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008555:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008558:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008568:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008572:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.008748:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008753:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.008763:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008766:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008774:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008777:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008784:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008787:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008795:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008798:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.008806:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008809:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008816:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008819:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008826:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008829:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.008867:0:326199:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.008871:0:326199:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431499.027981:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431499.028014:0:326204:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431499.028023:0:326204:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431499.028062:0:326204:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431499.028089:0:326204:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:3.0:1789431499.028109:0:326204:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:3.0:1789431499.028118:0:326204:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431499.028128:0:326204:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431499.028133:0:326204:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:3.0:1789431499.028142:0:326204:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c174bcb8 00010000:00010000:3.0:1789431499.028147:0:326204:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:3.0:1789431499.028157:0:326204:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:1.0:1789431499.028919:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431499.028932:0:326203:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431499.028938:0:326203:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431499.028953:0:326203:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431499.028966:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1789431499.028985:0:326203:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:1.0:1789431499.028992:0:326203:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431499.028998:0:326203:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431499.029002:0:326203:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:1.0:1789431499.029007:0:326203:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c160bcb8 00010000:00010000:1.0:1789431499.029010:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:1.0:1789431499.029019:0:326203:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:1.0:1789431499.030542:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030560:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.030572:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030576:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.030585:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030588:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.030596:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030599:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.030608:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030611:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.030620:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030624:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.030631:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030634:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.030643:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030646:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.030653:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.030664:0:326203:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2a67500, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.030672:0:326203:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.030698:0:326203:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libssl.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.030784:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.030797:0:326203:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.030811:0:326203:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.030827:0:326203:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913473920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.030870:0:326203:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.030898:0:326203:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913473920:192.168.203.128@tcp:101:openssl.0 00000100:00100000:1.0:1789431499.031156:0:326203:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc2180 going to sleep for 11 seconds 00000080:00200000:2.0:1789431499.031760:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.031777:0:326202:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.031798:0:326202:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.031824:0:326202:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.031879:0:326202:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:2.0:1789431499.031895:0:326202:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:2.0:1789431499.031904:0:326202:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.031913:0:326202:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.031917:0:326202:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:2.0:1789431499.031926:0:326202:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1593cb8 00010000:00010000:2.0:1789431499.031930:0:326202:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:2.0:1789431499.031940:0:326202:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1789431499.034131:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034151:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.034166:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034170:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.034179:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034182:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.034190:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034193:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.034200:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034203:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.034211:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034215:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.034222:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034225:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.034233:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034236:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.034245:0:326202:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.034249:0:326202:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431499.039008:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039031:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.039047:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039052:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039065:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039069:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039078:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039081:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039089:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039093:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039101:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039105:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039113:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039116:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039124:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039127:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039135:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039140:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431499.039339:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039345:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.039356:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039359:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039369:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039372:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039381:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039385:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039394:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039397:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039406:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039409:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039418:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039421:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039430:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039434:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039455:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039460:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:2.0:1789431499.039615:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039620:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.039629:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039633:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039643:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039646:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039655:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039658:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039667:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039671:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039680:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039683:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039693:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039696:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039706:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039710:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039717:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039721:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431499.039883:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039888:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.039897:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039900:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039907:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039910:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039917:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039920:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039927:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039930:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.039938:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039941:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039948:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039951:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039959:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039961:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.039968:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.039973:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431499.040105:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040111:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.040122:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040125:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040134:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040137:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040147:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040151:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040161:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040164:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040173:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040177:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040185:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040189:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040196:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040200:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040208:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040212:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431499.040335:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040340:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.040351:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040354:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040363:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040366:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040374:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040376:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040384:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040387:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.040396:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040399:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040409:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040412:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040421:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040424:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.040434:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.040439:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000100:00100000:0.0:1789431499.046736:0:326203:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913473920:192.168.203.128@tcp:101:openssl.0 00010000:00010000:0.0:1789431499.046772:0:326203:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.048772:0:326203:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.048783:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1789431499.049295:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049311:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.049326:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049331:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049341:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049345:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049353:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049356:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049366:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049368:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049376:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049380:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049388:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049391:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049399:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049402:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049426:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049431:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431499.049623:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049628:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.049638:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049641:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049649:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049652:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049661:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049665:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049672:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049675:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.049683:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049686:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049696:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049699:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049708:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049711:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.049722:0:326204:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.049726:0:326204:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00010000:00010000:0.0:1789431499.053071:0:326203:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053086:0:326203:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053101:0:326203:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ec00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431499.053108:0:326203:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053116:0:326203:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053127:0:326203:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053136:0:326203:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431499.053144:0:326203:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.053151:0:326203:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27e3fb lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431499.053191:0:326203:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913473920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431499.053210:0:326203:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431499.053323:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1789431499.053340:0:326203:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00010000:0.0:1789431499.053351:0:326203:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c160b9a0 00010000:00010000:0.0:1789431499.053356:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1789431499.053371:0:326203:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1789431499.054208:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054225:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.054240:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054244:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.054256:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054260:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.054272:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054276:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.054290:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054294:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.054302:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054306:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.054312:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054315:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.054321:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054323:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.054330:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.054342:0:326203:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2a52f00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431499.054350:0:326203:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431499.054382:0:326203:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypto.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431499.054473:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.054484:0:326203:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431499.054498:0:326203:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431499.054510:0:326203:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4700 x1876350913474176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431499.054520:0:326203:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431499.054547:0:326203:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913474176:192.168.203.128@tcp:101:openssl.0 00000100:00100000:1.0:1789431499.060784:0:326203:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397840 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.060870:0:326203:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913474176:192.168.203.128@tcp:101:openssl.0 00010000:00010000:1.0:1789431499.060896:0:326203:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060912:0:326203:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060920:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060931:0:326203:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060940:0:326203:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060946:0:326203:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.060952:0:326203:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060959:0:326203:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060967:0:326203:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060974:0:326203:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.060981:0:326203:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.060989:0:326203:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e402 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.061012:0:326203:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd4700 x1876350913474176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.061029:0:326203:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.061066:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1789431499.061082:0:326203:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00010000:1.0:1789431499.061093:0:326203:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c160b9a0 00010000:00010000:1.0:1789431499.061098:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1789431499.061107:0:326203:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:1.0:1789431499.061806:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061822:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.061870:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061875:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.061885:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061887:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.061898:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061901:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.061913:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061916:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.061925:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061929:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.061939:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061942:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.061953:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061956:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.061964:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.061979:0:326203:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8cd9800, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.061987:0:326203:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.062016:0:326203:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libz.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.062148:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.062159:0:326203:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.062172:0:326203:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.062182:0:326203:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913474432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.062192:0:326203:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.062219:0:326203:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913474432:192.168.203.128@tcp:101:openssl.0 00000100:00100000:1.0:1789431499.065866:0:326203:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.072109:0:326203:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326203:1876350913474432:192.168.203.128@tcp:101:openssl.0 00010000:00010000:1.0:1789431499.072142:0:326203:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072159:0:326203:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072168:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072176:0:326203:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072187:0:326203:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072197:0:326203:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.072203:0:326203:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072212:0:326203:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072220:0:326203:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072227:0:326203:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.072234:0:326203:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.072241:0:326203:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e409 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326203 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.072261:0:326203:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913474432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.072277:0:326203:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.072315:0:326203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1789431499.072329:0:326203:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00010000:1.0:1789431499.072340:0:326203:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c160b9a0 00010000:00010000:1.0:1789431499.072343:0:326203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:1.0:1789431499.072356:0:326203:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:1.0:1789431499.073683:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073701:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.073714:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073720:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.073728:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073732:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.073740:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073743:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.073751:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073755:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.073763:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.073767:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.073775:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074057:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074072:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074076:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074082:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074087:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431499.074273:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074278:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.074287:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074291:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074300:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074303:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074312:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074316:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074325:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074328:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074336:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074339:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074348:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074351:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074360:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074363:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074371:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074375:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431499.074689:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074696:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.074707:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074710:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074719:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074722:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074729:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074732:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074739:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074742:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.074749:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074752:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074760:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074763:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.074770:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.074774:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.075042:0:326203:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.075048:0:326203:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431499.106274:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.106304:0:326207:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.106315:0:326207:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.106359:0:326207:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.106386:0:326207:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1789431499.106417:0:326207:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:2.0:1789431499.106428:0:326207:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.106439:0:326207:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.106451:0:326207:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:2.0:1789431499.106461:0:326207:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c154bcb8 00010000:00010000:2.0:1789431499.106465:0:326207:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:2.0:1789431499.106476:0:326207:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:3.0:1789431499.111248:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111272:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.111288:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111292:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.111302:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111305:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.111314:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111317:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.111327:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111330:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.111340:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111344:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.111355:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111358:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.111367:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111370:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.111377:0:326207:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.111382:0:326207:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431499.112040:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431499.112056:0:326208:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431499.112065:0:326208:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431499.112102:0:326208:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431499.112128:0:326208:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:0.0:1789431499.112148:0:326208:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:0.0:1789431499.112157:0:326208:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431499.112166:0:326208:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431499.112170:0:326208:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:0.0:1789431499.112179:0:326208:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c178bcb8 00010000:00010000:0.0:1789431499.112184:0:326208:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:0.0:1789431499.112194:0:326208:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:0.0:1789431499.113788:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113811:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.113825:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113829:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.113871:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113875:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.113886:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113890:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.113900:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113903:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.113913:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113918:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.113928:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113931:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.113939:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113942:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.113949:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.113953:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:0.0:1789431499.114187:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114193:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.114202:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114205:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.114214:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114217:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.114226:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114230:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.114239:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114243:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.114253:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114257:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.114265:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114268:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.114277:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114280:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.114289:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.114293:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:0.0:1789431499.114441:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.116152:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.116173:0:326209:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.116181:0:326209:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.116218:0:326209:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.116242:0:326209:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1789431499.116262:0:326209:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:2.0:1789431499.116272:0:326209:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.116281:0:326209:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.116285:0:326209:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:2.0:1789431499.116294:0:326209:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1bebcb8 00010000:00010000:2.0:1789431499.116299:0:326209:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:2.0:1789431499.116310:0:326209:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:0.0:1789431499.119743:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.119775:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119781:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.119794:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119798:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.119808:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119810:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.119819:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119822:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.119853:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119859:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.119873:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119876:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.119889:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119892:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.119904:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.119909:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:0.0:1789431499.120147:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120153:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.120163:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120167:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.120176:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120179:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.120190:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120193:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.120202:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120205:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.120214:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120217:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.120226:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120229:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.120237:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120241:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.120251:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120255:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431499.120396:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.120401:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.122353:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122370:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.122385:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122390:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.122399:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122402:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.122558:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122564:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.122578:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122582:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.122590:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122594:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.122602:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122606:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.122615:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122619:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.122628:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.122632:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431499.122947:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.122955:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.122971:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.122974:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.122985:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.122988:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.122997:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123000:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.123008:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123012:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123020:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123023:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123032:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123034:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123044:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123048:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431499.123252:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123257:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.123266:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123269:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.123277:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123280:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.123287:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123290:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.123298:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123301:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.123310:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123313:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123322:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123325:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123334:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123337:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.123348:0:326208:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.123352:0:326208:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.125873:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125896:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.125912:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125917:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.125926:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125929:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.125938:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125941:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.125950:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125953:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.125962:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125967:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.125976:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125980:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.125992:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.125997:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.126008:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126012:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431499.126252:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126257:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.126268:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126271:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.126286:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126289:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.126299:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126302:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.126311:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126315:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.126323:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126327:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.126335:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126338:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.126347:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126351:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.126360:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.126364:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431499.128956:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.128977:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.128995:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129000:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129010:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129013:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129022:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129026:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129039:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129043:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129053:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129057:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.129066:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129069:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.129078:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129081:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.129091:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129096:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431499.129323:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129329:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.129340:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129343:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129352:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129356:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129363:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129366:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129374:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129377:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.129386:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129390:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.129398:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129401:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.129979:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.129988:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.130001:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130006:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431499.130189:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130195:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.130208:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130212:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.130222:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130225:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.130234:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130237:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.130247:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130250:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.130259:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130262:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.130270:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130273:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.130281:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130284:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.130292:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.130296:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431499.131735:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131755:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.131769:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131774:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.131783:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131786:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.131795:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131798:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.131806:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131809:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.131822:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131825:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.131862:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131866:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.131877:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131880:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.131891:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.131896:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.132155:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132160:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.132168:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132171:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.132179:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132183:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.132191:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132194:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.132202:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132205:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.132214:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132217:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.132225:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132228:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.132236:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132240:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.132246:0:326209:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.132250:0:326209:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:1.0:1789431499.157295:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431499.157321:0:326212:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431499.157331:0:326212:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431499.157369:0:326212:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431499.157394:0:326212:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1789431499.157414:0:326212:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:1.0:1789431499.157942:0:326212:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431499.157952:0:326212:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431499.157957:0:326212:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e34c 00000080:00010000:1.0:1789431499.157966:0:326212:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e34c from it ffffbc42c1593cb8 00010000:00010000:1.0:1789431499.157971:0:326212:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:1.0:1789431499.157983:0:326212:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1789431499.161271:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161294:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.161309:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161315:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.161324:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161327:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.161334:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161337:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.161345:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161347:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.161355:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161359:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.161367:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161371:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.161379:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161382:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.161389:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.161393:0:326211:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.161820:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161827:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.161939:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161945:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.161953:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161957:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.161964:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161967:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.161975:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161977:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.161984:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161987:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.161995:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.161997:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.162005:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.162008:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.162017:0:326212:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.162020:0:326212:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431499.165302:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.165340:0:326211:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431499.165345:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431499.165352:0:326211:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431499.165358:0:326211:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=pjdfstest_33b929bb3f8942212881bfa8bf01b95f 00010000:00010000:2.0:1789431499.165428:0:326211:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1789431499.165469:0:326211:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431499.165648:0:326211:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1789431499.165664:0:326211:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431499.165667:0:326211:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e34c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abd6c expref: -99 pid: 324996 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000020:00100000:2.0:1789431499.165688:0:326211:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431499.165703:0:326211:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913474688/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 592/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431499.165732:0:326211:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326211:1876350913474688:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431499.166822:0:326211:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525a80 going to sleep for 11 seconds 00000100:00100000:0.0:1789431499.170253:0:326211:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326211:1876350913474688:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431499.170335:0:326211:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:0.0:1789431499.170344:0:326211:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x692:0x0], updating i_size 42 00000080:00200000:2.0:1789431499.181438:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.181477:0:326214:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.181496:0:326214:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.181530:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.181550:0:326214:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431499.181629:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.181640:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.181653:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.181689:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913474816:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431499.181818:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.185007:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913474816:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431499.185044:0:326214:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185059:0:326214:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185074:0:326214:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185093:0:326214:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185107:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185117:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185125:0:326214:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.185137:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2101c00 x1876350913474816/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431499.185154:0:326214:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185170:0:326214:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431499.185179:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431499.185213:0:326214:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431499.185220:0:326214:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e410 00000080:00010000:3.0:1789431499.185226:0:326214:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431499.185230:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c13cb8 00010000:00010000:3.0:1789431499.185234:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.185249:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431499.190716:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.190742:0:326214:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431499.190748:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.190785:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.190906:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.190919:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.190930:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.190963:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913474944:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.191096:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34180 going to sleep for 11 seconds 00000100:00100000:0.0:1789431499.195293:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913474944:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431499.195329:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195347:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195355:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195363:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195379:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195387:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431499.195393:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195402:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195421:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195430:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431499.195438:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195454:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e417 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431499.195477:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6300 x1876350913474944/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431499.195505:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431499.195545:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431499.195560:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431499.195572:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c13970 00010000:00010000:0.0:1789431499.195578:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431499.195587:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431499.195700:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.195719:0:326214:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431499.195723:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431499.195746:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431499.195798:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195806:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.195816:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431499.196087:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475072:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431499.196186:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397900 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.201024:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475072:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.201058:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201081:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201091:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201100:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201111:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201118:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.201123:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201131:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201138:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201145:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.201154:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201161:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e41e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.201183:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6d80 x1876350913475072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.201206:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.201241:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431499.201254:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431499.201264:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c13970 00010000:00010000:1.0:1789431499.201268:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431499.201277:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431499.201372:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.201384:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.201398:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.201401:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.201424:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.201428:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.201437:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.201460:0:326214:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431499.201465:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431499.201487:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431499.201539:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201547:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.201560:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.201588:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475200:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.201694:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.206403:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475200:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.206461:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206478:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206486:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206503:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206514:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206520:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.206525:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206532:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206540:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206548:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.206556:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.206562:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e425 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.206586:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913475200/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.206604:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.206647:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431499.206662:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431499.206674:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c13970 00010000:00010000:1.0:1789431499.206678:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431499.206688:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431499.206797:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.206812:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.206828:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.206899:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.206913:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.206932:0:326214:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0082c00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.206940:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.206966:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.207033:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.207041:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.207053:0:326214:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.207061:0:326214:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913475456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.207070:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.207099:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475456:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.207338:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.211543:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475456:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.211580:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211597:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211606:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211614:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211625:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211632:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.211637:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211645:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211652:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211660:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.211668:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.211674:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e42c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.211698:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913475456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.211715:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.211755:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431499.211770:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431499.211781:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c139a0 00010000:00010000:1.0:1789431499.211785:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431499.211794:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431499.212677:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212693:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.212704:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212708:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.212716:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212719:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.212728:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212731:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.212738:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212741:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.212750:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212753:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.212761:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212764:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.212773:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212776:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.212785:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.212797:0:326214:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0082f00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.212804:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.212829:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.213002:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.213012:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.213024:0:326214:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.213034:0:326214:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913475712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.213045:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.213069:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475712:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.213348:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.217701:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475712:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.217734:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217751:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217759:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217768:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217777:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217783:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.217789:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217800:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217818:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217826:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.217877:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.217885:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e433 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.217909:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913475712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.217927:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.217966:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431499.217981:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431499.217991:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c139a0 00010000:00010000:1.0:1789431499.217995:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431499.218004:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431499.218497:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218512:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.218527:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218532:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.218541:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218544:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.218554:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218557:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.218565:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218568:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.218575:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218579:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.218587:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218591:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.218599:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218602:0:326214:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.218609:0:326214:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.218622:0:326214:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c3980, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.218628:0:326214:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.218655:0:326214:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.218719:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.218727:0:326214:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.218740:0:326214:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.218749:0:326214:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913475968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.218760:0:326214:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.218785:0:326214:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475968:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.219062:0:326214:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.224047:0:326214:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326214:1876350913475968:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.224080:0:326214:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224097:0:326214:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224105:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224113:0:326214:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224124:0:326214:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224132:0:326214:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.224138:0:326214:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224146:0:326214:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224153:0:326214:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224162:0:326214:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.224169:0:326214:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.224176:0:326214:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e43a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326214 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.224200:0:326214:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913475968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.224216:0:326214:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.224253:0:326214:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431499.224268:0:326214:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431499.224279:0:326214:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1c139a0 00010000:00010000:1.0:1789431499.224283:0:326214:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431499.224291:0:326214:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431499.234380:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431499.234420:0:326217:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431499.234430:0:326217:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431499.234469:0:326217:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431499.234502:0:326217:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431499.234521:0:326217:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431499.234532:0:326217:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431499.234543:0:326217:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431499.234547:0:326217:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e410 00000080:00010000:0.0:1789431499.234557:0:326217:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e410 from it ffffbc42c1ddbcb8 00010000:00010000:0.0:1789431499.234562:0:326217:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431499.234574:0:326217:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431499.236245:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236266:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431499.236283:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236288:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.236297:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236301:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.236310:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236313:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.236322:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236326:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431499.236334:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236339:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.236348:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236351:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.236361:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236364:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431499.236374:0:326217:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.236378:0:326217:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.238181:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238202:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.238216:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238221:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.238230:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238233:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.238240:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238244:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.238251:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238254:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.238263:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238267:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.238275:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238278:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.238288:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238291:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.238300:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.238305:0:326216:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.239179:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.239205:0:326216:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:3.0:1789431499.239210:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431499.239216:0:326216:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431499.239220:0:326216:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=pjdfstest_0f17e1bd12e0f4450588142807de1d35 00010000:00010000:3.0:1789431499.239278:0:326216:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431499.239310:0:326216:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431499.239946:0:326216:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431499.239959:0:326216:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431499.239962:0:326216:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e410 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abfbf expref: -99 pid: 326214 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1789431499.239989:0:326216:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431499.240005:0:326216:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7800 x1876350913476224/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 592/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431499.240044:0:326216:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326216:1876350913476224:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431499.240227:0:326216:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.243181:0:326216:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326216:1876350913476224:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431499.243249:0:326216:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.243257:0:326216:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x693:0x0], updating i_size 42 00000080:00200000:0.0:1789431499.250088:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431499.250115:0:326219:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431499.250125:0:326219:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431499.250158:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431499.250175:0:326219:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431499.250242:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.250252:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.250263:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431499.250295:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476352:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431499.250423:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431499.258664:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476352:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431499.258705:0:326219:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258721:0:326219:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258735:0:326219:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258748:0:326219:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258767:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258775:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.258782:0:326219:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431499.259946:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd5180 x1876350913476352/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431499.259967:0:326219:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.259984:0:326219:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431499.259992:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431499.260033:0:326219:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431499.260040:0:326219:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:0.0:1789431499.260046:0:326219:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431499.260050:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4bcb8 00010000:00010000:0.0:1789431499.260054:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.260065:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431499.263634:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.263661:0:326219:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431499.263667:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431499.263703:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431499.263785:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.263799:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.263811:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431499.264195:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476480:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431499.264606:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431499.272356:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476480:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431499.272389:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272405:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272483:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272492:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272508:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272515:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431499.272520:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272529:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272537:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272546:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431499.272553:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.272560:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e448 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431499.272583:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd5180 x1876350913476480/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431499.272600:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431499.272636:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431499.272651:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431499.272660:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b970 00010000:00010000:0.0:1789431499.272665:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431499.272673:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431499.272764:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431499.276087:0:326219:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431499.276105:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431499.276383:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431499.276550:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.276564:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.276575:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.276609:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476608:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.276813:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.280244:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476608:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.280277:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280293:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280301:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280309:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280319:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280328:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.280333:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280341:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280349:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280358:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.280365:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280371:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e44f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.280393:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913476608/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.280421:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.280467:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431499.280482:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431499.280493:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b970 00010000:00010000:1.0:1789431499.280497:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431499.280505:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431499.280609:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.280621:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.280635:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.280639:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.280649:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.280654:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.280664:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.280678:0:326219:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431499.280683:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431499.280703:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431499.280753:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280761:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.280772:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.281146:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476736:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.281299:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.285239:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476736:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.285270:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285287:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285295:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285304:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285314:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285322:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.285328:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285336:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285344:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285352:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.285360:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.285366:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e456 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.285388:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913476736/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.285482:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.285521:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431499.285536:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431499.285547:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b970 00010000:00010000:1.0:1789431499.285551:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431499.285566:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431499.285662:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.285673:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.285684:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.285687:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.285694:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.285711:0:326219:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c3c80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.285719:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.285740:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.286034:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.286044:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.286057:0:326219:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.286066:0:326219:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913476992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.286076:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.286100:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476992:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.286398:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.290390:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913476992:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.290510:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290527:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290535:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290544:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290554:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290560:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.290566:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290575:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290583:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290591:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.290598:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.290607:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e45d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.290638:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913476992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.290655:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.290694:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431499.290710:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431499.290721:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b9a0 00010000:00010000:1.0:1789431499.290725:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431499.290734:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431499.291519:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291534:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.291547:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291551:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.291560:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291563:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.291570:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291573:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.291582:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291585:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.291593:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291597:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.291609:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291612:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.291620:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291623:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.291631:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.291643:0:326219:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c4d15500, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.291650:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.291675:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.291739:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.291749:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.291762:0:326219:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.291772:0:326219:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913477248/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.296078:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.296134:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913477248:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.296389:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.300570:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913477248:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.300603:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300619:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300631:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300638:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300649:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300656:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.300661:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300669:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300676:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300684:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.300692:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.300699:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e464 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.300722:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913477248/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.300740:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.300775:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431499.300929:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431499.300939:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b9a0 00010000:00010000:1.0:1789431499.300944:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431499.300954:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431499.301496:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301507:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.301523:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301528:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.301536:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301540:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.301549:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301552:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.301562:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301565:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.301572:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301576:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.301582:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301585:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.301591:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301594:0:326219:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.301602:0:326219:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.301614:0:326219:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0082f00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431499.301621:0:326219:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431499.301645:0:326219:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431499.301706:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.301715:0:326219:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431499.301728:0:326219:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431499.301737:0:326219:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913477504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431499.301748:0:326219:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431499.301773:0:326219:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913477504:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431499.302249:0:326219:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.306178:0:326219:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326219:1876350913477504:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431499.306212:0:326219:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306229:0:326219:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306237:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306245:0:326219:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306259:0:326219:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306266:0:326219:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431499.306272:0:326219:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306280:0:326219:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306289:0:326219:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306297:0:326219:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431499.306305:0:326219:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.306312:0:326219:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e46b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326219 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.306338:0:326219:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913477504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431499.306355:0:326219:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431499.306390:0:326219:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431499.306483:0:326219:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431499.306496:0:326219:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c4b9a0 00010000:00010000:1.0:1789431499.306501:0:326219:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431499.306510:0:326219:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431499.315633:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315658:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.315673:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315678:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.315687:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315690:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.315699:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315702:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.315713:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315716:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.315725:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315730:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.315739:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315742:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.315754:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315757:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.315767:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.315772:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.316402:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.316423:0:326221:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=16 00000080:00200000:3.0:1789431499.316429:0:326221:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000080:00200000:1.0:1789431499.318035:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431499.318057:0:326222:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431499.318065:0:326222:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431499.318093:0:326222:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431499.318117:0:326222:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431499.318137:0:326222:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431499.318147:0:326222:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431499.318157:0:326222:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431499.318161:0:326222:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:1.0:1789431499.318169:0:326222:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c1bcb8 00010000:00010000:1.0:1789431499.318173:0:326222:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431499.318183:0:326222:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431499.319794:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319811:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.319824:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319828:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.319872:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319876:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.319883:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319886:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.319895:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319898:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.319906:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319909:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.319916:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319919:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.319926:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319929:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.319938:0:326222:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.319942:0:326222:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000002:00010000:2.0:1789431499.321607:0:326221:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_33b929bb3f8942212881bfa8bf01b95f,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431499.321698:0:326221:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.321710:0:326221:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.321722:0:326221:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.321754:0:326221:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477632:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431499.322118:0:326221:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c85256c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.330506:0:326221:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477632:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431499.330545:0:326221:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330576:0:326221:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x693:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330586:0:326221:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330601:0:326221:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330616:0:326221:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330632:0:326221:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330641:0:326221:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330649:0:326221:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.330663:0:326221:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d2101880 x1876350913477632/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431499.330679:0:326221:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330691:0:326221:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1683 (0 0) 00000080:00200000:2.0:1789431499.330699:0:326221:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431499.330725:0:326221:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x693:0x0], updating i_size 42 00000080:00200000:2.0:1789431499.330728:0:326221:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x693:0x0](ffff9811c4801148): rc = 0 00000080:00010000:2.0:1789431499.330733:0:326221:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x693:0x0](ffff9811c4801148) for lock 0x1ffb2ee1bd27e472 00000080:00010000:2.0:1789431499.330746:0:326221:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00010000:2.0:1789431499.330750:0:326221:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e472 from it ffffbc42c178bab0 00010000:00010000:2.0:1789431499.330754:0:326221:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.330764:0:326221:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431499.331139:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000100:00100000:2.0:1789431499.331204:0:326221:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477760:192.168.203.128@tcp:33:pjdfstest.0 00000100:00100000:2.0:1789431499.331314:0:326221:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c85256c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.333767:0:326221:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477760:192.168.203.128@tcp:33:pjdfstest.0 00000080:00200000:3.0:1789431499.333981:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.334013:0:326221:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=16 00000080:00200000:3.0:1789431499.334021:0:326221:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.334053:0:326221:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_0f17e1bd12e0f4450588142807de1d35,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.334124:0:326221:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.334136:0:326221:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.334146:0:326221:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326221 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.334171:0:326221:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477888:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431499.334309:0:326221:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34240 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.337966:0:326221:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913477888:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431499.338001:0:326221:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338031:0:326221:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x692:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338040:0:326221:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338055:0:326221:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338068:0:326221:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338089:0:326221:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338098:0:326221:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338106:0:326221:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.338121:0:326221:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff9811d1bc4380 x1876350913477888/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431499.338135:0:326221:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338147:0:326221:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1682 (0 0) 00000080:00200000:3.0:1789431499.338155:0:326221:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431499.338182:0:326221:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x692:0x0], updating i_size 42 00000080:00200000:3.0:1789431499.338186:0:326221:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x692:0x0](ffff9811c4872a08): rc = 0 00000080:00010000:3.0:1789431499.338192:0:326221:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x692:0x0](ffff9811c4872a08) for lock 0x1ffb2ee1bd27e479 00000080:00010000:3.0:1789431499.338206:0:326221:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00010000:3.0:1789431499.338210:0:326221:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e479 from it ffffbc42c178bab0 00010000:00010000:3.0:1789431499.338213:0:326221:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.338224:0:326221:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431499.338282:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000100:00100000:3.0:1789431499.338346:0:326221:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913478016:192.168.203.128@tcp:33:pjdfstest.0 00000100:00100000:3.0:1789431499.338456:0:326221:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34240 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.341071:0:326221:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326221:1876350913478016:192.168.203.128@tcp:33:pjdfstest.0 00000080:00200000:2.0:1789431499.341188:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341205:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341231:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341235:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341239:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341242:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341245:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341248:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341251:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341253:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341255:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341257:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341260:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341262:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341264:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341266:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341269:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341271:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341273:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341275:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341278:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341281:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341284:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341286:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341289:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341291:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341293:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341296:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341298:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341300:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341302:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341305:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341307:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341309:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341311:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341313:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341316:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341318:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341320:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341322:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341325:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341327:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341329:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341331:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341333:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341336:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341338:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341341:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341343:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341346:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341348:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341351:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341353:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341355:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341358:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341360:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341362:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341365:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341367:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341369:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341371:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341374:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341376:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341378:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341380:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341382:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341385:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341387:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341389:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341391:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341394:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341396:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341398:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341401:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341473:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341477:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341480:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341483:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341485:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341488:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341490:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341493:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341495:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341498:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341500:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341502:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341504:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341507:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341509:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341511:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341514:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341516:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341518:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341520:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341523:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341525:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341527:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341530:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341532:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341535:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341537:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341539:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341541:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341544:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341546:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341548:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341551:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341553:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341555:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341557:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.341559:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.341561:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341564:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.341566:0:326221:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.341568:0:326221:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.341570:0:326221:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.347748:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.347776:0:326224:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.347786:0:326224:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.347815:0:326224:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.347868:0:326224:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431499.347888:0:326224:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431499.347904:0:326224:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.347915:0:326224:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.347920:0:326224:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:2.0:1789431499.347930:0:326224:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c63cb8 00010000:00010000:2.0:1789431499.347934:0:326224:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431499.347944:0:326224:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431499.349605:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349627:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.349641:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349646:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.349654:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349657:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.349665:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349668:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.349676:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349679:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.349688:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349692:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.349700:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349704:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.349712:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349715:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.349722:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349727:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431499.349975:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349980:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.349988:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.349990:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.349997:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350000:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350006:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350008:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350015:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350017:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350023:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350026:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350033:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350036:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350045:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350047:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350055:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350059:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431499.350208:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350214:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.350223:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350226:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350233:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350236:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350243:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350246:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350253:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350255:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.350262:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350264:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350271:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350274:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350282:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350284:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.350291:0:326224:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.350294:0:326224:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431499.362389:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.362430:0:326227:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.362450:0:326227:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.362483:0:326227:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.362509:0:326227:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431499.362528:0:326227:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431499.362538:0:326227:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.362549:0:326227:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.362553:0:326227:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:2.0:1789431499.362563:0:326227:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c7bcb8 00010000:00010000:2.0:1789431499.362568:0:326227:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431499.362578:0:326227:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1789431499.364190:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364208:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.364220:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364224:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.364234:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364237:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.364244:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364248:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.364256:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364259:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.364268:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364272:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.364279:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364282:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.364290:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364294:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.364302:0:326227:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.364307:0:326227:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.368151:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368174:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.368188:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368193:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.368201:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368204:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.368212:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368215:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.368223:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368226:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.368234:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368239:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.368248:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368251:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.368261:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368265:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.368274:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.368278:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.369307:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369324:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.369331:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:3.0:1789431499.369337:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369341:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:3.0:1789431499.369347:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:3.0:1789431499.369351:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369354:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.369356:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:3.0:1789431499.369359:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369361:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:3.0:1789431499.369363:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:3.0:1789431499.369366:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369368:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.369370:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:3.0:1789431499.369372:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369374:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:3.0:1789431499.369377:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:3.0:1789431499.369379:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369381:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.369383:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:3.0:1789431499.369386:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369388:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:3.0:1789431499.369391:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:3.0:1789431499.369393:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369397:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.369400:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:3.0:1789431499.369546:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.369552:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369635:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369640:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369644:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369647:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369650:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369652:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369655:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369657:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369659:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369661:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369664:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369666:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369668:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369670:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369674:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369676:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369679:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369682:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369684:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369686:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369688:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369690:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369693:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369695:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369697:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369699:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369702:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369704:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369706:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369708:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369710:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369712:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369715:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369717:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369719:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369721:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369723:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369726:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369728:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369730:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369733:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369735:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369738:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369741:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369743:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369745:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369747:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369750:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369752:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369754:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369756:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369758:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369761:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369763:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369765:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369767:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369770:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.369772:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.369774:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369776:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.369778:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.369781:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.369783:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.370114:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.370118:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370122:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.370125:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.370128:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370130:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.370133:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.370135:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370138:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.370140:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.370142:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370144:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.370146:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.370149:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370151:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.370153:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.370155:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370158:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.370160:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.370162:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370164:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.370166:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.370169:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370171:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:2.0:1789431499.370173:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:2.0:1789431499.370176:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370178:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=16 00000080:00200000:2.0:1789431499.370180:0:326226:0:(symlink.c:141:ll_get_link()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.370182:0:326226:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.370186:0:326226:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=16 00000080:00200000:3.0:1789431499.375962:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431499.375989:0:326229:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431499.375999:0:326229:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431499.376029:0:326229:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431499.376052:0:326229:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431499.376072:0:326229:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431499.376080:0:326229:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431499.376089:0:326229:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431499.376093:0:326229:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:3.0:1789431499.376103:0:326229:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c53cb8 00010000:00010000:3.0:1789431499.376107:0:326229:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431499.376118:0:326229:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1789431499.377743:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377764:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.377777:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377782:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.377790:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377793:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.377802:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377805:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.377813:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377816:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.377825:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377830:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.377871:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377875:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.377884:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377887:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.377895:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.377899:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431499.378104:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378109:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.378119:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378124:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378132:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378135:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378143:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378146:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378154:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378156:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378164:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378167:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.378174:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378177:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.378186:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378189:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.378197:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378201:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.378353:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378357:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.378368:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378371:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378380:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378384:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378392:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378395:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378403:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378405:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.378413:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378416:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.378425:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.378429:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.378437:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.382294:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.382986:0:326229:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.383004:0:326229:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431499.392795:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.392825:0:326232:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.392865:0:326232:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.392896:0:326232:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.392920:0:326232:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431499.392940:0:326232:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:2.0:1789431499.392950:0:326232:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431499.392960:0:326232:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431499.392964:0:326232:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e441 00000080:00010000:2.0:1789431499.392972:0:326232:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e441 from it ffffbc42c1c83cb8 00010000:00010000:2.0:1789431499.392977:0:326232:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431499.392988:0:326232:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431499.399687:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399712:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.399727:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399732:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.399741:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399745:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.399752:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399755:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.399763:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399766:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.399774:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399778:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.399796:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399800:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.399809:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399812:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.399819:0:326232:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.399823:0:326232:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.400091:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400098:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.400252:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400257:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.400269:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400272:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.400279:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400282:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.400290:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400293:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.400302:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400305:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.400312:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400315:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.400322:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400326:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.400333:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.400336:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.402756:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.402794:0:326231:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, flags=0 00000080:00200000:1.0:1789431499.402801:0:326231:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431499.402807:0:326231:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_33b929bb3f8942212881bfa8bf01b95f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431499.403592:0:326231:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:1.0:1789431499.403629:0:326231:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431499.403696:0:326231:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.403749:0:326231:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.403758:0:326231:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:1.0:1789431499.403765:0:326231:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431499.403769:0:326231:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e472 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x693:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac0f3 expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.403797:0:326231:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e441 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac059 expref: -99 pid: 326219 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000020:00100000:1.0:1789431499.403824:0:326231:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431499.404548:0:326231:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913478272/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431499.404584:0:326231:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326231:1876350913478272:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431499.404792:0:326231:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.411355:0:326231:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326231:1876350913478272:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431499.411502:0:326231:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x693:0x0](ffff9811c4801148) 00000080:00200000:2.0:1789431499.426758:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431499.426799:0:326234:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431499.426811:0:326234:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431499.427661:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431499.427684:0:326234:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431499.427757:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.427768:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.427780:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.427969:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478400:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431499.428097:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525600 going to sleep for 11 seconds 00010000:00010000:0.0:1789431499.431006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431499.433498:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478400:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431499.433533:0:326234:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433550:0:326234:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433565:0:326234:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433579:0:326234:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433593:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433601:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433615:0:326234:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.433628:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2103b80 x1876350913478400/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431499.433644:0:326234:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433664:0:326234:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431499.433672:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431499.433705:0:326234:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431499.433712:0:326234:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e480 00000080:00010000:3.0:1789431499.433718:0:326234:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431499.433722:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154bcb8 00010000:00010000:3.0:1789431499.433725:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.433735:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431499.437081:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.437176:0:326234:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431499.437184:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.437220:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.437302:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.437316:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.437327:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.437360:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478528:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.437616:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.440805:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478528:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.441081:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441099:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441205:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441214:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441224:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441231:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.441236:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441243:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441253:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441261:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.441268:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.441275:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e487 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.441295:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913478528/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.441311:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.441424:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431499.441458:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431499.441469:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b970 00010000:00010000:2.0:1789431499.441473:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431499.441483:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431499.441569:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.441585:0:326234:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431499.441591:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431499.441610:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431499.441655:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.441663:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.441673:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.441696:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478656:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.445663:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.445722:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478656:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431499.445745:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.445763:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.445770:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446214:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446229:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446236:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431499.446241:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446249:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446257:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446265:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431499.446272:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446280:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e48e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.446301:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913478656/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431499.446323:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.446358:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431499.446372:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431499.446383:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b970 00010000:00010000:2.0:1789431499.446387:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431499.446397:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431499.446512:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.446523:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.446535:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.446539:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.446546:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.446549:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.446557:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.446569:0:326234:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431499.446574:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431499.446599:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431499.446660:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446669:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.446681:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.446704:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478784:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.446817:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.450385:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913478784:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431499.450427:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450454:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450462:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450470:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450479:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450487:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431499.450492:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450500:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450508:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450516:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431499.450524:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.450531:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e495 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.450553:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913478784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431499.450571:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.450610:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431499.450624:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431499.450634:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b970 00010000:00010000:2.0:1789431499.450638:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431499.450646:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431499.450739:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.450751:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.450768:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.450771:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.450977:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.450995:0:326234:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b63500, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431499.451003:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431499.451028:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431499.451089:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.451097:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431499.451348:0:326234:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431499.451360:0:326234:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100000 x1876350913479040/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431499.451371:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.451461:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479040:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.451720:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.455474:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479040:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431499.455506:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455522:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455530:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455538:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455550:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455557:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431499.455563:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455571:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455578:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455585:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431499.455594:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.455600:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e49c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.455622:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913479040/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431499.455639:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.455673:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431499.455687:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431499.455698:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b9a0 00010000:00010000:2.0:1789431499.455702:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431499.455711:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1789431499.457011:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457030:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.457046:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457051:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.457060:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457063:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.457074:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457076:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.457084:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457086:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.457094:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457097:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.457178:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457183:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.457195:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457198:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.457210:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.457223:0:326234:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5425b00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431499.457232:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431499.457263:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431499.457343:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.457354:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431499.457371:0:326234:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431499.457383:0:326234:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100000 x1876350913479296/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431499.457393:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.457498:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479296:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.457745:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.461454:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479296:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431499.461489:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461505:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461512:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461519:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461530:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461537:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431499.461542:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461552:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461560:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461568:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431499.461575:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.461581:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e4a3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.461603:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913479296/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431499.461620:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.461656:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431499.461670:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:2.0:1789431499.461681:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b9a0 00010000:00010000:2.0:1789431499.461686:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431499.461695:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431499.462290:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462305:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.462317:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462322:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.462332:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462335:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.462344:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462347:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.462354:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462357:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.462365:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462369:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.462379:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462382:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.462391:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462394:0:326234:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.462404:0:326234:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.462418:0:326234:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b63e00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431499.462425:0:326234:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431499.462461:0:326234:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431499.462526:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.462536:0:326234:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431499.462549:0:326234:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431499.462559:0:326234:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100000 x1876350913479552/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431499.462570:0:326234:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.462597:0:326234:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479552:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.462830:0:326234:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:2.0:1789431499.467262:0:326234:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326234:1876350913479552:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431499.467296:0:326234:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467313:0:326234:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467321:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467330:0:326234:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467341:0:326234:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467348:0:326234:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431499.467354:0:326234:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467362:0:326234:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467371:0:326234:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467379:0:326234:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431499.467388:0:326234:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.467395:0:326234:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e4aa lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326234 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431499.467430:0:326234:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913479552/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431499.467457:0:326234:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431499.467497:0:326234:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431499.467513:0:326234:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:2.0:1789431499.467524:0:326234:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c154b9a0 00010000:00010000:2.0:1789431499.467527:0:326234:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431499.467536:0:326234:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431499.477595:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431499.477622:0:326237:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431499.477631:0:326237:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431499.477659:0:326237:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431499.477682:0:326237:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431499.477701:0:326237:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431499.477710:0:326237:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431499.477719:0:326237:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431499.477723:0:326237:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e480 00000080:00010000:1.0:1789431499.477732:0:326237:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e480 from it ffffbc42c1c9bcb8 00010000:00010000:1.0:1789431499.477737:0:326237:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431499.477752:0:326237:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431499.479337:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479355:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431499.479369:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479373:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.479382:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479384:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.479393:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479396:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.479403:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479406:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431499.479413:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479417:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.479424:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479427:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.479435:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479438:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431499.479459:0:326237:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.479464:0:326237:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431499.480113:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.480129:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.480155:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.480160:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.480174:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.480991:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.481011:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481015:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.481025:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481028:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.481038:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481105:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.481120:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481123:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.481183:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481189:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.481204:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.481209:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431499.482462:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431499.482482:0:326236:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, flags=0 00000080:00200000:1.0:1789431499.482488:0:326236:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431499.482499:0:326236:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_0f17e1bd12e0f4450588142807de1d35, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431499.482555:0:326236:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431499.482599:0:326236:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431499.482661:0:326236:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.482715:0:326236:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.482723:0:326236:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431499.482730:0:326236:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431499.482733:0:326236:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e479 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x692:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac10f expref: -99 pid: 326221 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431499.482752:0:326236:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e480 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac132 expref: -99 pid: 326234 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431499.482771:0:326236:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431499.482788:0:326236:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf800 x1876350913479808/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431499.482819:0:326236:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326236:1876350913479808:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431499.482968:0:326236:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d1bc29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.488080:0:326236:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326236:1876350913479808:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431499.488166:0:326236:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x692:0x0](ffff9811c4872a08) 00000080:00200000:0.0:1789431499.496172:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431499.496201:0:326239:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431499.496210:0:326239:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431499.496242:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431499.496262:0:326239:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431499.496344:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.496356:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.496367:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431499.496400:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913479936:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431499.496537:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811e0397900 going to sleep for 11 seconds 00000100:00100000:1.0:1789431499.502019:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913479936:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431499.502055:0:326239:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502073:0:326239:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502088:0:326239:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502102:0:326239:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502127:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502137:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502146:0:326239:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431499.502161:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd5180 x1876350913479936/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431499.502178:0:326239:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502196:0:326239:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431499.502204:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431499.502240:0:326239:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431499.502247:0:326239:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431499.502254:0:326239:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431499.502258:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2bcb8 00010000:00010000:1.0:1789431499.502261:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431499.502273:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431499.504530:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.504558:0:326239:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431499.504564:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.504601:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.504684:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.504697:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.504708:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.504739:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480064:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.504877:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.509992:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480064:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.510024:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510041:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510048:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510056:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510072:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510079:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.510085:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510092:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510100:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510108:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.510116:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510123:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4b8 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.510145:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913480064/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.510164:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431499.510201:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431499.510215:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431499.510225:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b970 00010000:00010000:3.0:1789431499.510229:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431499.510237:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431499.510331:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.510361:0:326239:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431499.510365:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.510387:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.510456:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510464:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.510475:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.510886:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480192:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.511001:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.514377:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480192:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.514419:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514436:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514452:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514461:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514472:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514479:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.514484:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514500:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514508:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514515:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.514522:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.514528:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4bf lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.514550:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913480192/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.514566:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431499.514601:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431499.514614:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431499.514624:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b970 00010000:00010000:3.0:1789431499.514628:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431499.514638:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431499.514731:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.514742:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.514753:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.514756:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.514764:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.514767:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.514779:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.514793:0:326239:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431499.514798:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431499.514819:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431499.515056:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.515065:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.515075:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.515099:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480320:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.515240:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.524332:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480320:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.524373:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524392:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524409:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524418:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524428:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524435:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.524449:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524457:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524465:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524473:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.524481:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.524652:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e4c6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.524683:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913480320/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.524701:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431499.524740:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431499.524757:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431499.524768:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b970 00010000:00010000:3.0:1789431499.524773:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431499.524782:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431499.526332:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.526354:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.526369:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.526373:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.526381:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.526397:0:326239:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8ceef00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431499.526406:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431499.526450:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431499.526534:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.526550:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431499.526896:0:326239:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431499.526913:0:326239:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7b80 x1876350913480576/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431499.526925:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.526954:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480576:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.527212:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.536927:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480576:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.536973:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.536990:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.536998:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537005:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537016:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537022:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.537028:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537035:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537043:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537050:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.537058:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.537064:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e4cd lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.537088:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913480576/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.537104:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431499.537143:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431499.537158:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431499.537169:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:3.0:1789431499.537172:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431499.537181:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431499.538310:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538329:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431499.538465:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538472:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.538483:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538487:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.538497:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538501:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.538509:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538511:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431499.538520:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538524:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.538532:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538535:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.538545:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538548:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431499.538559:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431499.538582:0:326239:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8cee900, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431499.538590:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431499.538729:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431499.538809:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.538819:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431499.539152:0:326239:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431499.539166:0:326239:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7b80 x1876350913480832/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431499.539179:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431499.539206:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480832:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431499.539596:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a34540 going to sleep for 11 seconds 00000100:00100000:3.0:1789431499.549540:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913480832:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431499.549572:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549587:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549594:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549602:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549613:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549620:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431499.549626:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549633:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549642:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549651:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431499.549659:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431499.549665:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e4d4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431499.549688:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7b80 x1876350913480832/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431499.549706:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431499.549745:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431499.549759:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431499.549769:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:3.0:1789431499.549773:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431499.550110:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431499.550926:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.550942:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431499.550956:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.550961:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.550972:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.550976:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.550985:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.550989:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.550997:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.551000:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431499.551014:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.551018:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.551028:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.551031:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.551040:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.551043:0:326239:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431499.551051:0:326239:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431499.551068:0:326239:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62a80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431499.551075:0:326239:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431499.551107:0:326239:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431499.551183:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431499.551191:0:326239:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431499.551201:0:326239:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431499.551210:0:326239:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100000 x1876350913481088/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431499.551218:0:326239:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431499.551238:0:326239:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913481088:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431499.551457:0:326239:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000100:00100000:0.0:1789431499.560021:0:326239:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326239:1876350913481088:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431499.560055:0:326239:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560075:0:326239:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560083:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560091:0:326239:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560101:0:326239:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560107:0:326239:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431499.560113:0:326239:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560121:0:326239:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560130:0:326239:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560137:0:326239:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431499.560157:0:326239:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431499.560165:0:326239:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4db lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 326239 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431499.560189:0:326239:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913481088/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431515 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431499.560212:0:326239:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431499.560253:0:326239:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431499.560266:0:326239:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431499.560277:0:326239:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:0.0:1789431499.560281:0:326239:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431499.560290:0:326239:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431500.456040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431500.582416:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913481216:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431500.582550:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913481472:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431500.582594:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789431500.582600:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913481344:192.168.203.128@tcp:400:kworker.0 00010000:00080000:0.0:1789431500.599029:0:2390:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811c8cd5180 x1876350913481472/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431516 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431500.599055:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913481472:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431500.599097:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913481344:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431500.599110:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913481216:192.168.203.128@tcp:400:kworker.0 00000001:02000400:2.0:1789431501.052316:0:326371:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:3.0:1789431501.481987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431502.502704:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431502.687077:0:326510:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:1.0:1789431503.525977:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431504.550937:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431505.575582:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431505.702358:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913481600:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431505.702462:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913481856:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431505.702498:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789431505.702517:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913481728:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431505.707425:0:2392:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9811d1e45800: committing for last_committed 4294973995 gen 1 00000100:00100000:2.0:1789431505.707461:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913481728:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431505.707486:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913481600:192.168.203.128@tcp:400:kworker.0 00010000:00080000:2.0:1789431505.707501:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc6300 x1876350913481856/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431521 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431505.707512:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913481856:192.168.203.128@tcp:400:kworker.0 00010000:00010000:1.0:1789431506.598206:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431507.624357:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1789431508.633659:0:327087:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_08: symlink returns EEXIST if the name2 argument already exists ========================================================== 20:18:27 (1789431507) 00010000:00010000:2.0:1789431508.646998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431509.670034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431510.176759:0:327225:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/08.t against ext4 filesystem 00010000:00010000:3.0:1789431510.695161:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431510.822009:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913481984:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431510.822103:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913482240:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431510.822136:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431510.822143:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913482112:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431510.824064:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913481984:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431510.827821:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913482112:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431510.827861:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc4a80 x1876350913482240/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431526 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431510.827873:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913482240:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431511.721564:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431512.742388:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431512.866462:0:327487:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/08.t against lustre filesystem 00000080:00200000:1.0:1789431512.962762:0:327491:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431512.962799:0:327491:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431512.962820:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431512.962870:0:327491:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431512.962880:0:327491:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431512.962888:0:327491:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431512.962892:0:327491:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:1.0:1789431512.962899:0:327491:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1edbb58 00010000:00010000:1.0:1789431512.962903:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431512.962910:0:327491:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431512.962926:0:327491:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431512.962941:0:327491:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=13 00000080:00200000:1.0:1789431512.962946:0:327491:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:1.0:1789431512.962959:0:327491:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431512.963004:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327491 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431512.963012:0:327491:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327491 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431512.963021:0:327491:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327491 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431512.963047:0:327491:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327491:1876350913482368:192.168.203.128@tcp:101:lfs.0 00000100:00100000:1.0:1789431512.970787:0:327491:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:2.0:1789431512.987790:0:327491:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327491:1876350913482368:192.168.203.128@tcp:101:lfs.0 00010000:00010000:2.0:1789431512.987815:0:327491:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987853:0:327491:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987861:0:327491:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987872:0:327491:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987882:0:327491:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987900:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987907:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987914:0:327491:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431512.987925:0:327491:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdf480 x1876350913482368/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431528 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431512.987938:0:327491:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987948:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431512.987959:0:327491:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431512.987966:0:327491:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987971:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987976:0:327491:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431512.987996:0:327491:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1789431512.988002:0:327491:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431512.988027:0:327491:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1789431512.988031:0:327491:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:2.0:1789431512.988037:0:327491:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:2.0:1789431512.988050:0:327491:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431512.988054:0:327491:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1edbb20 00010000:00010000:2.0:1789431512.988057:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431512.988066:0:327491:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200008:2.0:1789431512.988104:0:327491:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1789431512.988111:0:327491:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431512.988130:0:327491:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431512.988139:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431512.988146:0:327491:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431512.988153:0:327491:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431512.988157:0:327491:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431512.988160:0:327491:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:2.0:1789431512.988163:0:327491:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1edbcb8 00010000:00010000:2.0:1789431512.988167:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431512.988174:0:327491:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200008:2.0:1789431512.988714:0:327491:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1789431512.988720:0:327491:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431512.988729:0:327491:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431512.988738:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431512.988748:0:327491:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431512.988754:0:327491:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431512.988759:0:327491:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431512.988761:0:327491:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:2.0:1789431512.988766:0:327491:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1edbcb8 00010000:00010000:2.0:1789431512.988769:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431512.988776:0:327491:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431512.988916:0:327491:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431512.988925:0:327491:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431512.988932:0:327491:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431512.988942:0:327491:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431512.988948:0:327491:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431512.988951:0:327491:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431512.988954:0:327491:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e384 00000080:00010000:2.0:1789431512.988958:0:327491:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e384 from it ffffbc42c1edbbb0 00010000:00010000:2.0:1789431512.988961:0:327491:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1789431512.988968:0:327491:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1789431512.988977:0:327491:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:2.0:1789431512.988983:0:327491:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:2.0:1789431512.988999:0:327491:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffdd136f220 00000080:00200000:2.0:1789431512.989010:0:327491:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:2.0:1789431512.989032:0:327491:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1789431512.989075:0:327491:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1789431512.989082:0:327491:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431512.989085:0:327491:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e384 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795abe14 expref: -99 pid: 325712 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:2.0:1789431512.989112:0:327491:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431512.989122:0:327491:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913482624/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431512.989143:0:327491:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327491:1876350913482624:192.168.203.128@tcp:36:lfs.0 00010000:00010000:3.0:1789431512.989183:0:6631:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1789431512.989206:0:6631:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989215:0:6631:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431512.989220:0:6631:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431512.989228:0:327491:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00010000:00010000:3.0:1789431512.989228:0:6631:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989241:0:6631:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989256:0:6631:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431512.989266:0:6631:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431512.989287:0:6631:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989293:0:6631:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431512.989319:0:6631:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989326:0:6631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e4e2 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac26d expref: -99 pid: 327491 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431512.989335:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431512.989337:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431512.989339:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431512.989365:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913482752:192.168.203.128@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1789431512.991920:0:327491:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327491:1876350913482624:192.168.203.128@tcp:36:lfs.0 00000080:00200008:0.0:1789431512.992028:0:327491:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000100:00100000:3.0:1789431512.995610:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913482752:192.168.203.128@tcp:103:ldlm_bl.0 00000080:00200008:3.0:1789431513.170327:0:326955:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431513.170341:0:326955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431513.170380:0:326955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431513.170457:0:326955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 326955 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.170470:0:326955:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 326955 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.170482:0:326955:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 326955 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431513.170513:0:326955:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326955:1876350913482880:192.168.203.128@tcp:101:bash.0 00000100:00100000:3.0:1789431513.170624:0:326955:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.175296:0:326955:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:326955:1876350913482880:192.168.203.128@tcp:101:bash.0 00010000:00010000:1.0:1789431513.175333:0:326955:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175357:0:326955:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175371:0:326955:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175384:0:326955:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175399:0:326955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175408:0:326955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175416:0:326955:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.175429:0:326955:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4a80 x1876350913482880/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431513.175444:0:326955:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175460:0:326955:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1789431513.175467:0:326955:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431513.175506:0:326955:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431513.175513:0:326955:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431513.175520:0:326955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431513.175523:0:326955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d53cb8 00010000:00010000:1.0:1789431513.175527:0:326955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.175536:0:326955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431513.175607:0:326955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431513.175628:0:326955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431513.175639:0:326955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431513.175650:0:326955:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431513.175658:0:326955:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431513.175663:0:326955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431513.175666:0:326955:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431513.175670:0:326955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d53b58 00010000:00010000:1.0:1789431513.175673:0:326955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431513.175681:0:326955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431513.175690:0:326955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.175698:0:326955:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431513.175710:0:326955:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431513.175714:0:326955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431513.175729:0:326955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431513.175738:0:326955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431513.175746:0:326955:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:1.0:1789431513.175753:0:326955:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431513.175757:0:326955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431513.175759:0:326955:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431513.175763:0:326955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0d53cb8 00010000:00010000:1.0:1789431513.175765:0:326955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431513.175772:0:326955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:1.0:1789431513.175794:0:326955:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431513.175800:0:326955:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431513.175807:0:326955:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431513.175815:0:326955:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431513.175823:0:326955:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431513.175826:0:326955:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431513.175829:0:326955:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431513.175878:0:326955:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d53c18 00010000:00010000:1.0:1789431513.175881:0:326955:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431513.175889:0:326955:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431513.175897:0:326955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.175902:0:326955:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:1.0:1789431513.175907:0:326955:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:2.0:1789431513.179544:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179565:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431513.179579:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179583:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.179591:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179594:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.179602:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179604:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.179613:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179616:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.179626:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179630:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.179639:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179641:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.179650:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179653:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.179661:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.179673:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0dcca80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431513.179683:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431513.179717:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431513.179809:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.179821:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431513.179863:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431513.179880:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913483136/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431513.179891:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431513.179917:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483136:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431513.180335:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431513.185116:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483136:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431513.185149:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185166:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185174:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185181:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185191:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185199:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431513.185204:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185211:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185224:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185231:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431513.185238:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.185245:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4f0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431513.185265:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101f80 x1876350913483136/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431513.185280:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431513.185314:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431513.185326:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00010000:2.0:1789431513.185333:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:2.0:1789431513.185336:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:2.0:1789431513.185356:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431513.185822:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185880:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431513.185896:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185901:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.185914:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185917:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.185924:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185927:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.185933:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185936:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.185943:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185947:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.185956:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185959:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.185967:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185970:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.185977:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.185980:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431513.186128:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186134:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431513.186143:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186147:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.186156:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186158:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.186166:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186169:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.186176:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186179:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.186185:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186188:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.186195:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186198:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.186206:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186209:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.186216:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.186229:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d1be0780, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431513.186236:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431513.186264:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libresolv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431513.186332:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.186341:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431513.186366:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431513.186376:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913483392/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431513.186385:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431513.186409:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483392:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431513.186642:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431513.192271:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483392:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431513.192304:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192397:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192405:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192413:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192423:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192430:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431513.192435:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192443:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192452:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192459:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431513.192465:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.192472:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e4f7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431513.192497:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101f80 x1876350913483392/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431513.192513:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431513.192548:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431513.192562:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00010000:2.0:1789431513.192572:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:2.0:1789431513.192576:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431513.192585:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1789431513.194107:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194126:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431513.194142:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194147:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.194155:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194157:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.194164:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194167:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.194174:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194177:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431513.194185:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194189:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.194200:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194203:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.194213:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.194215:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431513.196018:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.196050:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libdl.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2f58d80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431513.196060:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libdl.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431513.196097:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libdl.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431513.196190:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.196204:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431513.196219:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431513.196250:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913483648/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431513.196260:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431513.196367:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483648:192.168.203.128@tcp:101:prove.0 00000100:00100000:0.0:1789431513.196908:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd180 going to sleep for 11 seconds 00000100:00100000:0.0:1789431513.207368:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483648:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431513.207403:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207419:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207427:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207436:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207448:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207674:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431513.207679:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207688:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207698:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207705:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431513.207714:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.207721:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e4fe lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431513.207748:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101f80 x1876350913483648/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431513.207766:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431513.207804:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1789431513.207819:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00010000:0.0:1789431513.207830:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:0.0:1789431513.209038:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:0.0:1789431513.209055:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1789431513.210805:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210828:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431513.210880:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210884:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.210893:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210896:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.210904:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210907:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.210915:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210919:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.210930:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210934:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.210944:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210948:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.210960:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210964:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.210974:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.210989:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libm.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0080480, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431513.210998:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libm.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431513.211033:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libm.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431513.211154:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.211165:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431513.211181:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431513.211197:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913483904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431513.211216:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.211253:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483904:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.212101:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.220618:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913483904:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.220655:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220673:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220681:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220696:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220708:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220716:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.220722:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220730:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220738:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220747:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.220753:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.220760:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e505 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.220795:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913483904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.220814:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.220917:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1789431513.220933:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00010000:1.0:1789431513.220945:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:1.0:1789431513.220950:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1789431513.220965:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1789431513.221776:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221793:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431513.221807:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221811:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.221822:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221825:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.221890:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221894:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.221905:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221908:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.221918:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221922:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.221931:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221934:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.221943:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221946:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.221953:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.221966:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0080c00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431513.221973:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431513.222001:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431513.222072:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.222081:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431513.222094:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431513.222104:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913484160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431513.222113:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.222139:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484160:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.222933:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.225452:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484160:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.225482:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225497:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225504:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225511:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225520:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225526:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.225530:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225542:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225550:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225557:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.225564:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.225571:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e50c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.225594:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913484160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.225612:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.225649:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1789431513.225662:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00010000:1.0:1789431513.225673:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:1.0:1789431513.225677:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1789431513.225686:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1789431513.226450:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226464:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431513.226476:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226481:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.226489:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226492:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.226500:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226502:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.226510:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226513:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.226521:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226525:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.226533:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226536:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.226547:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226550:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.226558:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.226570:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d311d680, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431513.226577:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431513.226603:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libutil.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431513.227027:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.227039:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431513.227054:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431513.227068:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913484416/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431513.227080:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.227108:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484416:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.227377:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.232876:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484416:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.232908:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232924:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232933:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232941:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232952:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232960:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.232965:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232973:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232981:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.232990:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.232998:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.233006:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e513 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.233030:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913484416/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.233047:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.233085:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1789431513.233098:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00010000:1.0:1789431513.233109:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:1.0:1789431513.233113:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:1.0:1789431513.233121:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1789431513.233618:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233631:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431513.233645:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233649:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.233658:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233661:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.233669:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233672:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.233681:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233684:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431513.233694:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233698:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.233708:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233711:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.233719:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233722:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431513.233733:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.233737:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431513.517645:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517668:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431513.517683:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517762:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431513.517777:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517781:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431513.517789:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517793:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431513.517803:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517806:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431513.517815:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517819:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431513.517830:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517866:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431513.517880:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517883:0:327502:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431513.517891:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431513.517906:0:327502:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c121b080, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431513.517917:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431513.517956:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: librt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431513.518050:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.518062:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431513.518076:0:327502:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431513.518090:0:327502:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4700 x1876350913484672/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431513.518101:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431513.518129:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484672:192.168.203.128@tcp:101:prove.0 00000100:00100000:3.0:1789431513.518471:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431513.522495:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484672:192.168.203.128@tcp:101:prove.0 00010000:00010000:3.0:1789431513.522531:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522546:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522553:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522559:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522570:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522582:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431513.522587:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522594:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522601:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522608:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431513.522615:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431513.522621:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e51a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431513.522644:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4700 x1876350913484672/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431513.522661:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431513.522772:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431513.523051:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00010000:3.0:1789431513.523063:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db39a0 00010000:00010000:3.0:1789431513.523067:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:3.0:1789431513.523083:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1789431513.578483:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431513.578529:0:327502:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431513.578535:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431513.578573:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .proverc,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431513.578647:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.578660:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431513.578676:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431513.578709:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484800:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431513.578885:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.582706:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484800:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.582741:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582759:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582766:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582775:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582786:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582793:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.582798:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582808:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582817:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582825:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.582866:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.582873:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e521 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.582895:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2103480 x1876350913484800/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.582914:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.582954:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1789431513.582969:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00010000:1.0:1789431513.582980:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db3930 00010000:00010000:1.0:1789431513.582985:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1789431513.583117:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1789431513.590949:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.591098:0:327502:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:1.0:1789431513.591105:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431513.591143:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .prove,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431513.591294:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.591308:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.591320:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.591362:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484928:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.591573:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.598098:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913484928:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.598130:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598146:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598154:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598162:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598176:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598182:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.598188:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598195:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598212:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598220:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.598227:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.598364:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e528 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.598390:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913484928/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.598407:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.598446:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1789431513.598461:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00010000:1.0:1789431513.598470:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db3930 00010000:00010000:1.0:1789431513.598475:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1789431513.598484:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1789431513.647570:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.647590:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.647614:0:327502:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:1.0:1789431513.647620:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431513.647779:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: testrules.yml,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431513.647986:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.648000:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.648012:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.648044:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913485056:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.648180:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.651392:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913485056:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.651423:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651439:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651451:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651459:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651470:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651477:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.651483:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651491:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651499:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651507:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.651515:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.651521:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e52f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.651544:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913485056/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.651562:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.651601:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1789431513.651617:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00010000:1.0:1789431513.651628:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db3930 00010000:00010000:1.0:1789431513.651632:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1789431513.651642:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1789431513.656000:0:327502:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431513.656033:0:327502:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431513.656039:0:327502:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431513.656072:0:327502:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: t,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431513.656146:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.656156:0:327502:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.656169:0:327502:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.656203:0:327502:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913485184:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431513.656588:0:327502:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431513.661653:0:327502:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327502:1876350913485184:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431513.661684:0:327502:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661701:0:327502:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661709:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661718:0:327502:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661728:0:327502:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661735:0:327502:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431513.661740:0:327502:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661747:0:327502:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661756:0:327502:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661764:0:327502:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431513.661771:0:327502:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.661778:0:327502:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e536 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327502 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431513.661798:0:327502:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913485184/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431529 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431513.661820:0:327502:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431513.661926:0:327502:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1789431513.661940:0:327502:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00010000:1.0:1789431513.661949:0:327502:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1db38d0 00010000:00010000:1.0:1789431513.661953:0:327502:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1789431513.661961:0:327502:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1789431513.766339:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431513.899519:0:324946:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431513.899545:0:324946:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431513.899686:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913485312:192.168.203.128@tcp:63:ll_sa.0 00010000:00010000:0.0:1789431513.908674:0:2389:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908700:0:2389:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908708:0:2389:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908716:0:2389:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908727:0:2389:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908734:0:2389:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431513.908739:0:2389:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908747:0:2389:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908755:0:2389:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908763:0:2389:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431513.908771:0:2389:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431513.908778:0:2389:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f200/0x1ffb2ee1bd27e53d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324946 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431513.908819:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913485312:192.168.203.128@tcp:63:ll_sa.0 00000080:00200000:2.0:1789431514.000201:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.000225:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.000239:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.000244:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.001911:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001925:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.001936:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001940:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.001953:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001956:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.001967:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001971:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.001980:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001983:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.001996:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.001999:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.002008:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002013:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:2.0:1789431514.002777:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002790:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.002803:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002806:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.002815:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002818:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.002873:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002878:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.002892:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002896:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.002904:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002908:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.002914:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002917:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.002923:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002926:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.002934:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.002938:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431514.003110:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003116:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.003124:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003127:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.003134:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003137:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.003144:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003159:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.003170:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003174:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.003182:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003185:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.003194:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003196:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.003205:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003208:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.003217:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.003221:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:2.0:1789431514.004022:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004037:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.004049:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004053:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.004062:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004065:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.004073:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004076:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.004083:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004086:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.004093:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004096:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.004104:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004107:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.004115:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004117:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.004126:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.004130:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431514.006199:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006221:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.006235:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006239:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.006247:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006250:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.006259:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006263:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.006271:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006273:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.006282:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006286:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.006292:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006295:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.006302:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006305:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.006313:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.006317:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431514.010604:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010626:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.010658:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010665:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010676:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010680:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010688:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010691:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010698:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010702:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010710:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010714:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.010722:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010725:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.010732:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010734:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.010741:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010745:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:2.0:1789431514.010929:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010934:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.010947:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010950:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010959:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010963:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010972:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010974:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010982:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010986:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.010993:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.010996:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011003:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011007:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011020:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011023:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011031:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011034:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:2.0:1789431514.011190:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011196:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.011208:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011212:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.011220:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011223:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.011230:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011233:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.011241:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011245:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.011253:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011256:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011264:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011267:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011275:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011278:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.011286:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.011290:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.028642:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028667:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.028681:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028686:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.028696:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028700:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.028707:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028712:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.028720:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028723:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.028750:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028755:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.028763:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028766:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.028775:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028778:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.028786:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.028798:0:327503:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libtinfo.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0080d80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.028807:0:327503:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libtinfo.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.028917:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libtinfo.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.029150:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.029163:0:327503:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.029178:0:327503:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.029195:0:327503:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913485568/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.029207:0:327503:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.029235:0:327503:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327503:1876350913485568:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431514.029566:0:327503:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9540 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.034522:0:327503:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327503:1876350913485568:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431514.034554:0:327503:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034572:0:327503:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034580:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034589:0:327503:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034599:0:327503:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034606:0:327503:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.034611:0:327503:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034618:0:327503:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034626:0:327503:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034632:0:327503:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.034639:0:327503:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.034647:0:327503:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e544 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327503 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.034670:0:327503:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913485568/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sh.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.034703:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.034743:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1789431514.034755:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00010000:1.0:1789431514.034765:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0d4b9a0 00010000:00010000:1.0:1789431514.034769:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1789431514.034776:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1789431514.035618:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.035631:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.035642:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.035646:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.035655:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.035658:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.035667:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.035670:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.035678:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.035681:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.036096:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.036109:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.036128:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.036133:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.036150:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.036155:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.036170:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.036177:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431514.037595:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.037622:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.037648:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.037656:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.037676:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038148:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.038178:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038188:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.038211:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038216:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.038233:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038239:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.038259:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038265:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.038280:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038285:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.038469:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.038480:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.043443:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431514.043493:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.043521:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.043544:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.043555:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.043567:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431514.043572:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431514.043584:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d4bb58 00010000:00010000:1.0:1789431514.043590:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.043602:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.043626:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.043638:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431514.043653:0:327503:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.043662:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.043681:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.043716:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1789431514.043729:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1789431514.043738:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.043744:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.043748:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.043754:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0d4bcb8 00010000:00010000:1.0:1789431514.043759:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1789431514.043770:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1789431514.043816:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.043825:0:327503:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.043871:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.043886:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.043897:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1789431514.043909:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1789431514.043918:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.043925:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.043929:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.043934:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0d4bcb8 00010000:00010000:1.0:1789431514.043940:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1789431514.043949:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200008:1.0:1789431514.043986:0:327503:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.043992:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431514.044001:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.044008:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.044021:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.044027:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.044031:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431514.044033:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431514.044037:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d4bcb8 00010000:00010000:1.0:1789431514.044040:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.044047:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.044062:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431514.044068:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.044074:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.044082:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.044088:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.044091:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431514.044093:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431514.044096:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0d4bb58 00010000:00010000:1.0:1789431514.044099:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.044106:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.044113:0:327503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.044116:0:327503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431514.044121:0:327503:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.044124:0:327503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.044132:0:327503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.044137:0:327503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1789431514.044143:0:327503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:1.0:1789431514.044149:0:327503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.044152:0:327503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.044155:0:327503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.044157:0:327503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0d4bcb8 00010000:00010000:1.0:1789431514.044160:0:327503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1789431514.044167:0:327503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1789431514.064970:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.065002:0:327504:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.065012:0:327504:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.065054:0:327504:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.065080:0:327504:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1789431514.065098:0:327504:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1789431514.065107:0:327504:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431514.065117:0:327504:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.065121:0:327504:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:2.0:1789431514.065131:0:327504:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1edbcb8 00010000:00010000:2.0:1789431514.065136:0:327504:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1789431514.065146:0:327504:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1789431514.073602:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073629:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.073705:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073711:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.073723:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073727:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.073738:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073741:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.073752:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073755:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.073763:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073767:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.073783:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073786:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.073795:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073798:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.073807:0:327504:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.073813:0:327504:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.098631:0:327507:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431514.098661:0:327507:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431514.098671:0:327507:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431514.098993:0:327507:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431514.099020:0:327507:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1789431514.099041:0:327507:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1789431514.099050:0:327507:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431514.099058:0:327507:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431514.099062:0:327507:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:0.0:1789431514.099072:0:327507:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2bcb8 00010000:00010000:0.0:1789431514.099077:0:327507:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1789431514.099087:0:327507:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:3.0:1789431514.113162:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113189:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.113214:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113221:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.113238:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113244:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.113253:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113257:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.113268:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113271:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.113281:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113285:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.113294:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113296:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.113304:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113307:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.113315:0:327508:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.113319:0:327508:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.123238:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.123268:0:327511:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.123279:0:327511:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.123322:0:327511:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.123358:0:327511:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1789431514.123376:0:327511:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1789431514.123386:0:327511:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.123397:0:327511:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.123402:0:327511:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.123411:0:327511:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1453cb8 00010000:00010000:1.0:1789431514.123416:0:327511:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1789431514.123426:0:327511:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1789431514.125023:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431514.125051:0:327510:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431514.125062:0:327510:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431514.125101:0:327510:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431514.125128:0:327510:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:0.0:1789431514.125157:0:327510:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1789431514.125169:0:327510:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431514.125181:0:327510:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431514.125187:0:327510:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:0.0:1789431514.125199:0:327510:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e33cb8 00010000:00010000:0.0:1789431514.125206:0:327510:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:0.0:1789431514.125219:0:327510:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1789431514.127099:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127118:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.127132:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127136:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.127146:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127149:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.127158:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127161:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.127171:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127175:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.127185:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127189:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.127198:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127202:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.127211:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127214:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.127221:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.127224:0:327510:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431514.128032:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431514.128048:0:327512:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431514.128056:0:327512:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431514.128090:0:327512:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431514.128117:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431514.128142:0:327512:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1789431514.128151:0:327512:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431514.128164:0:327512:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431514.128168:0:327512:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:3.0:1789431514.128177:0:327512:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c148bcb8 00010000:00010000:3.0:1789431514.128181:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431514.128190:0:327512:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:3.0:1789431514.130170:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130195:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.130219:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130225:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.130242:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130248:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.130263:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130268:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.130285:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130292:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.130308:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130315:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.130332:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130338:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.130371:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130377:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.130394:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.130412:0:327512:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0f380, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431514.130421:0:327512:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431514.130456:0:327512:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libsigsegv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431514.130558:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431514.130572:0:327512:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431514.130589:0:327512:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431514.130607:0:327512:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4700 x1876350913485824/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431514.130621:0:327512:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431514.130655:0:327512:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913485824:192.168.203.128@tcp:101:awk.0 00000100:00100000:3.0:1789431514.130994:0:327512:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000080:00200000:1.0:1789431514.137162:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137185:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.137200:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137205:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.137213:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137216:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.137222:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137225:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.137233:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137236:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.137243:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137248:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.137258:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137261:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.137274:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137278:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.137286:0:327511:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.137290:0:327511:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.139649:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.139676:0:327510:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.141485:0:327510:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.141538:0:327510:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.141566:0:327510:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1789431514.141584:0:327510:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1789431514.141593:0:327510:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.141604:0:327510:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.141608:0:327510:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.141616:0:327510:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e33cb8 00010000:00010000:1.0:1789431514.141621:0:327510:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1789431514.141630:0:327510:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:1.0:1789431514.141701:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.141709:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:1.0:1789431514.141720:0:327510:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:1.0:1789431514.141762:0:327510:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:1.0:1789431514.141785:0:327510:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:1.0:1789431514.144045:0:327510:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.144065:0:327510:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431514.144107:0:327510:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.144135:0:327510:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.144158:0:327510:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.144170:0:327510:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.144181:0:327510:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431514.144187:0:327510:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431514.144197:0:327510:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0e33cb8 00010000:00010000:1.0:1789431514.144204:0:327510:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.144217:0:327510:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.144264:0:327510:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.144276:0:327510:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:1.0:1789431514.144284:0:327510:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3873, max_age 3887 00800000:00000004:1.0:1789431514.144425:0:327510:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3873, max_age 3887 00000100:00100000:1.0:1789431514.144496:0:327510:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327510:1876350913485952:192.168.203.128@tcp:41:df.0 00000100:00100000:1.0:1789431514.144641:0:327510:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.146620:0:327512:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913485824:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431514.146658:0:327512:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146678:0:327512:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146707:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146717:0:327512:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146729:0:327512:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146739:0:327512:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.146746:0:327512:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146755:0:327512:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146766:0:327512:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146776:0:327512:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.146788:0:327512:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.146797:0:327512:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119400/0x1ffb2ee1bd27e54b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.146822:0:327512:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4700 x1876350913485824/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.146890:0:327512:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.146938:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431514.146955:0:327512:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00010000:0.0:1789431514.146969:0:327512:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c148b9a0 00010000:00010000:0.0:1789431514.146975:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431514.146986:0:327512:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1789431514.147520:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147537:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.147561:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147568:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.147584:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147590:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.147605:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147611:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.147627:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147633:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.147648:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147655:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.147674:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147680:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.147695:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147702:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.147721:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.147740:0:327512:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0c26000, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.147751:0:327512:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.147784:0:327512:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libreadline.so.7,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.147914:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.147926:0:327512:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.147943:0:327512:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.147958:0:327512:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5500 x1876350913486208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.147972:0:327512:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.148005:0:327512:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486208:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431514.148279:0:327512:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd840 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.154922:0:327512:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486208:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431514.154953:0:327512:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.154970:0:327512:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.154978:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.154986:0:327512:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.154995:0:327512:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.155002:0:327512:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.155007:0:327512:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.155014:0:327512:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.155022:0:327512:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.155028:0:327512:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.155036:0:327512:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.155042:0:327512:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e552 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.155062:0:327512:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5500 x1876350913486208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.155083:0:327512:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.155119:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1789431514.155133:0:327512:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00010000:0.0:1789431514.155145:0:327512:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c148b9a0 00010000:00010000:0.0:1789431514.155149:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1789431514.155157:0:327512:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1789431514.155639:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155654:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.155674:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155681:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.155723:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155729:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.155747:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155752:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.155768:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155774:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.155788:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155795:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.155808:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155813:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.155828:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155883:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.155905:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.155925:0:327512:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c4d13b00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.155935:0:327512:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.155969:0:327512:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libmpfr.so.4,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.156048:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.156058:0:327512:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.156071:0:327512:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.156082:0:327512:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5500 x1876350913486464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.156092:0:327512:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.156119:0:327512:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486464:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431514.156385:0:327512:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd840 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.159405:0:327510:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327510:1876350913485952:192.168.203.128@tcp:41:df.0 00800000:00000004:1.0:1789431514.159438:0:327510:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:1.0:1789431514.159445:0:327510:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:1.0:1789431514.159450:0:327510:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000100:00100000:2.0:1789431514.161045:0:327512:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486464:192.168.203.128@tcp:101:awk.0 00010000:00010000:2.0:1789431514.161071:0:327512:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161086:0:327512:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161094:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161102:0:327512:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161112:0:327512:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161118:0:327512:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431514.161123:0:327512:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161130:0:327512:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161138:0:327512:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161151:0:327512:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431514.161157:0:327512:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.161164:0:327512:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2200/0x1ffb2ee1bd27e559 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431514.161181:0:327512:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5500 x1876350913486464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431514.161197:0:327512:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431514.161231:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1789431514.161243:0:327512:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00010000:2.0:1789431514.161253:0:327512:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c148b9a0 00010000:00010000:2.0:1789431514.161257:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:2.0:1789431514.161265:0:327512:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:2.0:1789431514.161666:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.161678:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.162485:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162497:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.162508:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162512:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.162522:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162525:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.162534:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162537:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.162545:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162550:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.162558:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162561:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.162569:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162572:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.162579:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.162594:0:327512:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0ef00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431514.162604:0:327512:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431514.162634:0:327512:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libgmp.so.10,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431514.163042:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431514.163054:0:327512:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431514.163071:0:327512:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431514.163085:0:327512:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6680 x1876350913486720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431514.163096:0:327512:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431514.163127:0:327512:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486720:192.168.203.128@tcp:101:awk.0 00000100:00100000:3.0:1789431514.163436:0:327512:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.168424:0:327512:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327512:1876350913486720:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431514.168454:0:327512:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168469:0:327512:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168476:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168485:0:327512:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168495:0:327512:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168502:0:327512:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.168508:0:327512:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168516:0:327512:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168524:0:327512:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168532:0:327512:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.168539:0:327512:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.168546:0:327512:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119600/0x1ffb2ee1bd27e560 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327512 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.168569:0:327512:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc6680 x1876350913486720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.168586:0:327512:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.168624:0:327512:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1789431514.168639:0:327512:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00010000:0.0:1789431514.168650:0:327512:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c148b9a0 00010000:00010000:0.0:1789431514.168654:0:327512:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1789431514.168663:0:327512:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1789431514.170190:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170211:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.170225:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170229:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.170237:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170240:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.170246:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170249:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.170257:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170260:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.170269:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170273:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.170281:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170285:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.170293:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170296:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.170747:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.170759:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431514.172900:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.172924:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.172947:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.172953:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.172969:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.172973:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.172988:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.172994:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173006:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173010:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173019:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173023:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173030:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173032:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173040:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173043:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173051:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173056:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431514.173291:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173297:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.173627:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173633:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173644:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173647:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173657:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173660:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173668:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173671:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.173690:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173694:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173704:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173707:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173717:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173720:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.173728:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.173732:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.174243:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174251:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.174262:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174265:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.174273:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174276:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.174284:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174287:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.174295:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174298:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.174321:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174325:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.174334:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174337:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.174356:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174360:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.174369:0:327512:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.174374:0:327512:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:0.0:1789431514.215620:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431514.215651:0:327515:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431514.215661:0:327515:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431514.215696:0:327515:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431514.215721:0:327515:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1789431514.215739:0:327515:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1789431514.215749:0:327515:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431514.215759:0:327515:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431514.215763:0:327515:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:0.0:1789431514.215772:0:327515:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1eabcb8 00010000:00010000:0.0:1789431514.215776:0:327515:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1789431514.215787:0:327515:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1789431514.216947:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.216962:0:327516:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.216970:0:327516:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.216991:0:327516:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.217007:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1789431514.217021:0:327516:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:2.0:1789431514.217028:0:327516:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431514.217034:0:327516:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.217037:0:327516:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:2.0:1789431514.217043:0:327516:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2bcb8 00010000:00010000:2.0:1789431514.217047:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:2.0:1789431514.217057:0:327516:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1789431514.217628:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217638:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.217649:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217653:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.217660:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217663:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.217671:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217674:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.217681:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217684:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.217693:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217697:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.217708:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217711:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.217720:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217723:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.217735:0:327515:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.217739:0:327515:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.218702:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.218725:0:327517:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.218735:0:327517:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.218770:0:327517:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.218797:0:327517:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1789431514.218816:0:327517:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:1.0:1789431514.218827:0:327517:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.218872:0:327517:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.218878:0:327517:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:1.0:1789431514.218889:0:327517:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1503cb8 00010000:00010000:1.0:1789431514.218895:0:327517:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1789431514.218908:0:327517:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:2.0:1789431514.221087:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221767:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.221791:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221795:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.221802:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221805:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.221812:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221815:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.221822:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221825:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.221848:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221851:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.221861:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221864:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.221870:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221873:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.221880:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.221894:0:327516:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c3e00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431514.221902:0:327516:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431514.221933:0:327516:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libssl.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431514.222017:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.222028:0:327516:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431514.222042:0:327516:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431514.222057:0:327516:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101500 x1876350913486976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431514.222069:0:327516:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431514.222759:0:327516:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913486976:192.168.203.128@tcp:101:openssl.0 00000100:00100000:2.0:1789431514.223057:0:327516:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000080:00200000:1.0:1789431514.223460:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223475:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.223495:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223502:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.223518:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223523:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.223537:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223541:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.223557:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223561:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.223574:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223579:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.223594:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223599:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.223613:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223619:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.223634:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.223639:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431514.225641:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225666:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.225711:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225719:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.225738:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225743:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.225760:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225765:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.225780:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225785:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.225802:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225808:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.225827:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225851:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.225864:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225867:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.225876:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.225880:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431514.226133:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226139:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.226149:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226156:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.226170:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226175:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.226430:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226437:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.226461:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226466:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.226484:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226490:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.226509:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226515:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.226533:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226538:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.226553:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.226560:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431514.227337:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227364:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.227392:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227397:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.227418:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227422:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.227433:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227435:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.227445:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227448:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.227458:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227462:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.227483:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227490:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.227512:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227517:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.227538:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.227545:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431514.229580:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229607:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.229630:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229637:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.229654:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229660:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.229678:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229706:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.229726:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229733:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.229748:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229754:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.229772:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229778:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.229795:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229801:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.229818:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.229825:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431514.230087:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230430:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.230447:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230451:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230459:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230462:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230471:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230474:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230483:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230486:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230495:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230500:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.230509:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230511:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.230519:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230522:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.230531:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230535:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431514.230943:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230949:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.230961:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230965:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230972:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230975:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230984:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230986:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.230994:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.230996:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.231004:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231008:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231016:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231019:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231029:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231032:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231040:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231043:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.231515:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231524:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.231535:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231539:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.231547:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231550:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.231559:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231562:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.231570:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231573:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.231583:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231587:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231595:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231598:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231607:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231610:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.231617:0:327517:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.231621:0:327517:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000100:00100000:0.0:1789431514.236592:0:327516:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913486976:192.168.203.128@tcp:101:openssl.0 00010000:00010000:0.0:1789431514.236625:0:327516:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236641:0:327516:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236650:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236658:0:327516:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236670:0:327516:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236677:0:327516:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.236682:0:327516:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236689:0:327516:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236697:0:327516:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236710:0:327516:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.236718:0:327516:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.236726:0:327516:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e567 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.236748:0:327516:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101500 x1876350913486976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.236766:0:327516:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.236802:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1789431514.236816:0:327516:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00010000:0.0:1789431514.236826:0:327516:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:0.0:1789431514.236863:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1789431514.236873:0:327516:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1789431514.238409:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238431:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.238447:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238452:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.238462:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238466:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.238476:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238480:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.238487:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238491:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.238504:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238508:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.238521:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238524:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.238534:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238536:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.238622:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.238697:0:327516:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e1757200, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.238707:0:327516:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.238741:0:327516:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypto.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.238827:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.238863:0:327516:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.238879:0:327516:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.238894:0:327516:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7480 x1876350913487232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.238905:0:327516:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.238934:0:327516:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913487232:192.168.203.128@tcp:101:openssl.0 00000100:00100000:0.0:1789431514.239326:0:327516:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.244032:0:327516:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913487232:192.168.203.128@tcp:101:openssl.0 00010000:00010000:0.0:1789431514.244066:0:327516:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244083:0:327516:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244091:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244099:0:327516:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244109:0:327516:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244116:0:327516:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.244121:0:327516:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244130:0:327516:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244139:0:327516:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244147:0:327516:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.244156:0:327516:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.244163:0:327516:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e56e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.244190:0:327516:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7480 x1876350913487232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.244208:0:327516:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.244244:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1789431514.244267:0:327516:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00010000:0.0:1789431514.244277:0:327516:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:0.0:1789431514.244282:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1789431514.244291:0:327516:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1789431514.245016:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245030:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.245046:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245050:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.245058:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245060:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.245068:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245072:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.245080:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245083:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.245093:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245097:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.245107:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245111:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.245120:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245123:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.245130:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.245143:0:327516:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2a52f00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.245149:0:327516:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.245177:0:327516:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libz.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.245399:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.245410:0:327516:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.245544:0:327516:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.245555:0:327516:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7480 x1876350913487488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.245567:0:327516:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.245592:0:327516:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913487488:192.168.203.128@tcp:101:openssl.0 00000100:00100000:0.0:1789431514.245994:0:327516:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.253454:0:327516:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327516:1876350913487488:192.168.203.128@tcp:101:openssl.0 00010000:00010000:0.0:1789431514.253489:0:327516:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253506:0:327516:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253519:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253526:0:327516:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253537:0:327516:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253544:0:327516:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.253549:0:327516:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253557:0:327516:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253564:0:327516:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253572:0:327516:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.253579:0:327516:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.253586:0:327516:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e575 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327516 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.253608:0:327516:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7480 x1876350913487488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.253624:0:327516:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.253659:0:327516:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1789431514.253675:0:327516:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00010000:0.0:1789431514.253684:0:327516:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c0e2b9a0 00010000:00010000:0.0:1789431514.253699:0:327516:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1789431514.253708:0:327516:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:0.0:1789431514.254189:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254201:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.254214:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254218:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254227:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254230:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254243:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254246:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254255:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254257:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254266:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254270:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254279:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254283:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254291:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254294:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254302:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254306:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431514.254460:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254465:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.254474:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254476:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254483:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254486:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254500:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254503:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254511:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254514:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254522:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254525:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254533:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254536:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254545:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254548:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254554:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254559:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431514.254698:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254703:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.254712:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254714:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254721:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254724:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254734:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254737:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254747:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254749:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.254758:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254761:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254769:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254772:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254779:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254782:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.254789:0:327516:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.254793:0:327516:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431514.274671:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431514.274930:0:327520:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431514.274940:0:327520:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431514.274977:0:327520:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431514.275001:0:327520:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431514.275019:0:327520:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:3.0:1789431514.275028:0:327520:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431514.275040:0:327520:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431514.275045:0:327520:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e4b1 00000080:00010000:3.0:1789431514.275054:0:327520:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4b1 from it ffffbc42c1ed3cb8 00010000:00010000:3.0:1789431514.275058:0:327520:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431514.275069:0:327520:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:0.0:1789431514.276320:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276320:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276337:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.276337:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.276363:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276363:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276368:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.276368:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.276377:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276377:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276380:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.276380:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.276388:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276388:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276391:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.276391:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.276399:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276399:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276402:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.276402:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.276409:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276410:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276413:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276413:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276421:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276422:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276424:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276424:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276432:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276433:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276435:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276435:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.276442:0:327520:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.276444:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.276446:0:327520:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.276447:0:327519:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.276993:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.277012:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.277018:0:327519:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8194600, open_flags 80c0, mode 81a4 opened 0 00000080:00200000:0.0:1789431514.277029:0:327519:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open|creat 00000002:00010000:0.0:1789431514.277066:0:327519:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x200000402:0x695:0x0]) in obj [0x200000402:0x1:0x0], intent: open|creat flags 0100301 00010000:00010000:0.0:1789431514.277091:0:327519:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:0.0:1789431514.277123:0:327519:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431514.277277:0:327519:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:0.0:1789431514.277285:0:327519:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431514.277289:0:327519:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e4b1 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac1d3 expref: -99 pid: 326239 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:0.0:1789431514.277317:0:327519:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 3/0,1 mode: --/CW res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327519 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.277325:0:327519:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 3/0,1 mode: --/CW res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327519 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.282790:0:327519:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.282823:0:327519:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7480 x1876350913487744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.282858:0:327519:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 3/0,1 mode: --/CW res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327519 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.283209:0:327519:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327519:1876350913487744:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:1.0:1789431514.283436:0:327519:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9540 going to sleep for 11 seconds 00000100:00100000:2.0:1789431514.294323:0:327519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327519:1876350913487744:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431514.294370:0:327519:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/0,1 mode: --/CW res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294401:0:327519:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/0,1 mode: --/PR res: [0x200000402:0x695:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294417:0:327519:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/0,1 mode: --/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294431:0:327519:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294444:0:327519:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294461:0:327519:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294469:0:327519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294478:0:327519:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431514.294491:0:327519:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9811c8cd7480 x1876350913487744/t4294973996(4294973996) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431530 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1789431514.294505:0:327519:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811c8cd7480 x1876350913487744/t4294973996(4294973996) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431530 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431514.294519:0:327519:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294531:0:327519:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1685 (0 0) 00000080:00200000:2.0:1789431514.294538:0:327519:0:(namei.c:945:ll_intent_lock()) intent lock 3 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431514.294569:0:327519:0:(namei.c:1621:ll_create_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open|creat 00000080:00200000:2.0:1789431514.294610:0:327519:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x695:0x0](ffff9811c4807448) 00000080:00200000:2.0:1789431514.294618:0:327519:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x695:0x0], updating i_size 0 00000080:00200000:2.0:1789431514.294664:0:327519:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x695:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:2.0:1789431514.294669:0:327519:0:(namei.c:1587:ll_create_node()) setting l_ast_data to inode [0x200000402:0x695:0x0](ffff9811c4807448) 00000080:00010000:2.0:1789431514.294673:0:327519:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x695:0x0](ffff9811c4807448) for lock 0x1ffb2ee1bd27e57c 00000080:00010000:2.0:1789431514.294684:0:327519:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x695:0x0](ffff9811c4807448) 00000080:00200008:2.0:1789431514.294700:0:327519:0:(file.c:1090:ll_file_open()) START file pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729:[0x200000402:0x695:0x0](ffff9811c4807448), flags 100300 00000002:00100000:2.0:1789431514.294724:0:327519:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811c8cd7480 x1876350913487744/t4294973996(4294973996) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431530 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1789431514.294768:0:327519:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e57c from it ffff9811d0a984e0 00010000:00010000:2.0:1789431514.294773:0:327519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.294784:0:327519:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431514.294814:0:327519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x695:0x0](ffff9811c4807448), inode mode 81a4 mask 40 00000080:00200008:2.0:1789431514.294899:0:327519:0:(file.c:479:ll_file_release()) START file pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729:[0x200000402:0x695:0x0](ffff9811c4807448), flags 100000 00000002:00100000:2.0:1789431514.294959:0:327519:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811c8cd7480 x1876350913487744/t4294973996(4294973996) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1789431514.294971:0:327519:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1789431514.294977:0:327519:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100e00 x1876350913488000/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1789431514.295005:0:327519:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327519:1876350913488000:192.168.203.128@tcp:35:pjdfstest.0 00000100:00100000:2.0:1789431514.295124:0:327519:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431514.299604:0:327519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327519:1876350913488000:192.168.203.128@tcp:35:pjdfstest.0 00000002:00100000:2.0:1789431514.299656:0:327519:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811c8cd7480 x1876350913487744/t4294973996(4294973996) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.304429:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.304459:0:327522:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.304469:0:327522:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.304502:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.304522:0:327522:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431514.304596:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.304608:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.304620:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.304654:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488128:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431514.304787:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.313596:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488128:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431514.313640:0:327522:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313658:0:327522:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313678:0:327522:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313696:0:327522:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313717:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313730:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313741:0:327522:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.313760:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bde680 x1876350913488128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431514.313808:0:327522:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313881:0:327522:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431514.313892:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431514.313935:0:327522:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431514.313947:0:327522:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e583 00000080:00010000:1.0:1789431514.313957:0:327522:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.313962:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c73cb8 00010000:00010000:1.0:1789431514.313967:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.313980:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431514.318450:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.318487:0:327522:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.318496:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.318541:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.318643:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.318658:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.318673:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.318733:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488256:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.319587:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.328723:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488256:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.328754:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328770:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328791:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328799:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328808:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328816:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.328821:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.328829:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.329040:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.329048:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.329056:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.329063:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e58a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.329091:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913488256/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.329107:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.329145:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.329158:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431514.329168:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c73970 00010000:00010000:1.0:1789431514.329172:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.329181:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431514.329273:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.329292:0:327522:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.329297:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.329391:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.329440:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.329448:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.329458:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.329484:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488384:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.329594:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.333562:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488384:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.333599:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333621:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333632:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333643:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333657:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333665:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.333672:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333682:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333692:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333701:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.333734:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.333743:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e591 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.333773:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913488384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.333814:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.333953:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.333973:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431514.333987:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c73970 00010000:00010000:1.0:1789431514.333994:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.334008:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.334122:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.334137:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.334162:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.334169:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.334185:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.334190:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.334206:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.334223:0:327522:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.334231:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.334257:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.334319:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.334327:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.334346:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.334372:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488512:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.334472:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.340981:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488512:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.341011:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.341766:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.341975:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.341984:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.341994:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342001:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.342008:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342016:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342034:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342043:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.342052:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342059:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e598 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.342090:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913488512/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.342108:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.342146:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.342161:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431514.342171:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c73970 00010000:00010000:1.0:1789431514.342175:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.342184:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.342282:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.342294:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.342309:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.342313:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.342322:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.342337:0:327522:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64d80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.342353:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.342378:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.342436:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.342443:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.342455:0:327522:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.342462:0:327522:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913488768/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.342471:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.342493:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488768:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.344155:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf96c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.353539:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913488768:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.353571:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353588:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353597:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353604:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353614:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353621:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.353626:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353635:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353648:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353655:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.353662:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.353669:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e59f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.353694:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913488768/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.353713:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.353758:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.353777:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431514.353787:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c739a0 00010000:00010000:1.0:1789431514.353792:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.353801:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431514.354703:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354718:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.354731:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354736:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.354746:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354752:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.354769:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354775:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.354784:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354788:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.354796:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354800:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.354809:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354812:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.354820:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354823:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.354884:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.354906:0:327522:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8194d80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.354917:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.354953:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.355041:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.355051:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.355063:0:327522:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.355074:0:327522:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5180 x1876350913489024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.355084:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.355114:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913489024:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431514.355382:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.359246:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913489024:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431514.359286:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359305:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359329:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359337:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359357:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359364:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.359370:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359378:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359386:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359396:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.359405:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.359415:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5a6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.359443:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5180 x1876350913489024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.359461:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.359498:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431514.359516:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:0.0:1789431514.359531:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c739a0 00010000:00010000:0.0:1789431514.359539:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431514.359552:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431514.360155:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360168:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.360181:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360185:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.360193:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360196:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.360203:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360206:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.360216:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360219:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.360226:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360229:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.360237:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360240:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.360249:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360252:0:327522:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.360260:0:327522:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.360273:0:327522:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf9980, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.360279:0:327522:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.360303:0:327522:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.360379:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.360388:0:327522:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.360401:0:327522:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.360410:0:327522:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5180 x1876350913489280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.360420:0:327522:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.360450:0:327522:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913489280:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431514.360676:0:327522:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.365768:0:327522:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327522:1876350913489280:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431514.365810:0:327522:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365828:0:327522:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365866:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365873:0:327522:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365882:0:327522:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365890:0:327522:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.365895:0:327522:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365902:0:327522:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365909:0:327522:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365917:0:327522:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.365924:0:327522:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.365930:0:327522:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5ad lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327522 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.365954:0:327522:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd5180 x1876350913489280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.365970:0:327522:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.366007:0:327522:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431514.366022:0:327522:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431514.366033:0:327522:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1c739a0 00010000:00010000:0.0:1789431514.366037:0:327522:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431514.366047:0:327522:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.380171:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.380200:0:327525:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.380211:0:327525:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.380247:0:327525:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.380279:0:327525:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.380395:0:327525:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.380407:0:327525:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.380417:0:327525:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.380421:0:327525:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e583 00000080:00010000:1.0:1789431514.380430:0:327525:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431514.380435:0:327525:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.380451:0:327525:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431514.381924:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.381946:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.381959:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.381964:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.381972:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.381976:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.382774:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382786:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.382803:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382807:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.382817:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382820:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.382830:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382860:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.382876:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382881:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.382891:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.382895:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.383524:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383536:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.383547:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383551:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.383559:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383562:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.383569:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383572:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.383579:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383582:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.383590:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383593:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.383602:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383605:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.383613:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383616:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.383625:0:327525:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.383628:0:327525:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431514.384543:0:327524:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.384637:0:327524:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, flags=1536 00000080:00200000:1.0:1789431514.396295:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.400234:0:327527:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.400249:0:327527:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.400289:0:327527:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.400332:0:327527:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431514.400365:0:327527:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431514.400375:0:327527:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.400385:0:327527:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.400389:0:327527:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e583 00000080:00010000:1.0:1789431514.400400:0:327527:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e583 from it ffffbc42c1577cb8 00010000:00010000:1.0:1789431514.400407:0:327527:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431514.400419:0:327527:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431514.407085:0:324993:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.407112:0:324993:0:(ldlm_request.c:1275:ldlm_cli_lock_create_pack()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.407242:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913489408:192.168.203.128@tcp:63:ll_sa.0 00000080:00200000:3.0:1789431514.408583:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408605:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.408630:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408636:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.408652:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408659:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.408677:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408683:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.408698:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408704:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.408718:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408725:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.408738:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408744:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.408760:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408766:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.408784:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.408791:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431514.409889:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.409908:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.409929:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.409937:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.409953:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.409959:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.409975:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.409980:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.409998:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410003:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.410015:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410018:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410027:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410030:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410039:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410042:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410052:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410056:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431514.410312:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410320:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.410331:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410334:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.410357:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410361:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.410371:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410374:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.410384:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410387:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.410397:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410402:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410416:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410419:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410428:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410431:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.410439:0:327527:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.410442:0:327527:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00010000:00010000:0.0:1789431514.422034:0:2390:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (FAILED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x0 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422057:0:2390:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422066:0:2390:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422075:0:2390:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422087:0:2390:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422095:0:2390:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.422102:0:2390:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422118:0:2390:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x206400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422128:0:2390:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x20e400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422138:0:2390:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.422150:0:2390:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.422160:0:2390:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e400/0x1ffb2ee1bd27e5b4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x3/0x3 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 324993 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.422206:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913489408:192.168.203.128@tcp:63:ll_sa.0 00000080:00200000:2.0:1789431514.428823:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430632:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430652:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.430665:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430669:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.430680:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430684:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.430693:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430697:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.430705:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430708:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.430717:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430722:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.430732:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430735:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.430747:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430752:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.430762:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.430768:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.431638:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.431655:0:327529:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, flags=0 00000080:00200000:0.0:1789431514.431661:0:327529:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.431667:0:327529:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:0.0:1789431514.431732:0:327529:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431514.431769:0:327529:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431514.431891:0:327529:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431514.431904:0:327529:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x695:0x0]: losing layout lock 00010000:00010000:0.0:1789431514.431956:0:327529:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431514.431964:0:327529:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431514.431970:0:327529:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1789431514.431973:0:327529:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e57c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x695:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ac43b expref: -99 pid: 327519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431514.431989:0:327529:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f400/0x1ffb2ee1bd27e583 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac450 expref: -99 pid: 327522 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000020:00100000:0.0:1789431514.432015:0:327529:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431514.432030:0:327529:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5180 x1876350913489664/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431514.432058:0:327529:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327529:1876350913489664:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431514.432172:0:327529:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000080:00200008:2.0:1789431514.432502:0:327530:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.432510:0:327530:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.432526:0:327530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.432536:0:327530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431514.432571:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.432579:0:327530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.432589:0:327530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431514.432604:0:327530:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913489792:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431514.432650:0:327530:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.438892:0:327529:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327529:1876350913489664:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431514.438988:0:327529:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x695:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789431514.439003:0:327529:0:(vvp_io.c:305:vvp_io_fini()) [0x200000402:0x695:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789431514.439017:0:327529:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x695:0x0](ffff9811c4807448) 00000100:00100000:2.0:1789431514.440326:0:327530:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913489792:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431514.440369:0:327530:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440385:0:327530:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440399:0:327530:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440411:0:327530:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440425:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440435:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440444:0:327530:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431514.440457:0:327530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2102a00 x1876350913489792/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431514.440472:0:327530:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440491:0:327530:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431514.440501:0:327530:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431514.440542:0:327530:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431514.440554:0:327530:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e5bb 00000080:00010000:2.0:1789431514.440563:0:327530:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.440568:0:327530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c1593cb8 00010000:00010000:2.0:1789431514.440572:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.440585:0:327530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431514.445680:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.445708:0:327530:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431514.445714:0:327530:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431514.445746:0:327530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431514.445829:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.446104:0:327530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.446124:0:327530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.446169:0:327530:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913489920:192.168.203.128@tcp:101:tail.0 00000100:00100000:0.0:1789431514.446588:0:327530:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.451583:0:327530:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913489920:192.168.203.128@tcp:101:tail.0 00010000:00010000:0.0:1789431514.451615:0:327530:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451632:0:327530:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451638:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451645:0:327530:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451655:0:327530:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451662:0:327530:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.451667:0:327530:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451674:0:327530:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451683:0:327530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451691:0:327530:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.451699:0:327530:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.451705:0:327530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e5c2 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.451730:0:327530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6d80 x1876350913489920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.451748:0:327530:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.451790:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431514.451805:0:327530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431514.451815:0:327530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c1593970 00010000:00010000:0.0:1789431514.451819:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431514.452268:0:327530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431514.452391:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.452410:0:327530:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431514.452415:0:327530:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431514.452438:0:327530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431514.452491:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.452499:0:327530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.452509:0:327530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.452534:0:327530:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490048:192.168.203.128@tcp:101:tail.0 00000100:00100000:0.0:1789431514.452732:0:327530:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.459792:0:327530:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490048:192.168.203.128@tcp:101:tail.0 00010000:00010000:0.0:1789431514.459826:0:327530:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459916:0:327530:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459924:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459932:0:327530:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459942:0:327530:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459950:0:327530:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.459956:0:327530:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459965:0:327530:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459974:0:327530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.459983:0:327530:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.459993:0:327530:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.460001:0:327530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5c9 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.460023:0:327530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6d80 x1876350913490048/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.460039:0:327530:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.460075:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431514.460089:0:327530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1789431514.460099:0:327530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c1593970 00010000:00010000:0.0:1789431514.460103:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431514.460113:0:327530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431514.460214:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.460226:0:327530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.460242:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.460246:0:327530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.460258:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.460262:0:327530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.460280:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.460293:0:327530:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431514.460539:0:327530:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431514.460570:0:327530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431514.460638:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.460651:0:327530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.460664:0:327530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.460694:0:327530:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490176:192.168.203.128@tcp:101:tail.0 00000100:00100000:0.0:1789431514.460974:0:327530:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.468595:0:327530:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490176:192.168.203.128@tcp:101:tail.0 00010000:00010000:0.0:1789431514.468628:0:327530:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468645:0:327530:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468652:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468660:0:327530:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468671:0:327530:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468678:0:327530:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.468684:0:327530:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468692:0:327530:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468700:0:327530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468707:0:327530:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.468715:0:327530:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.468723:0:327530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e5d0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.468746:0:327530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6d80 x1876350913490176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.468764:0:327530:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.468803:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431514.468818:0:327530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1789431514.468866:0:327530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c1593970 00010000:00010000:0.0:1789431514.468872:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431514.468881:0:327530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431514.468978:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.468990:0:327530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.469004:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.469008:0:327530:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.469017:0:327530:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.469031:0:327530:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8195b00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.469038:0:327530:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.469062:0:327530:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.469118:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.469125:0:327530:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.469136:0:327530:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.469143:0:327530:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6d80 x1876350913490432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.469153:0:327530:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.469177:0:327530:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490432:192.168.203.128@tcp:101:tail.0 00000100:00100000:0.0:1789431514.470829:0:327530:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.473812:0:327530:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327530:1876350913490432:192.168.203.128@tcp:101:tail.0 00010000:00010000:0.0:1789431514.474108:0:327530:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474129:0:327530:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474141:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474151:0:327530:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474163:0:327530:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474171:0:327530:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.474178:0:327530:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474189:0:327530:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474200:0:327530:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474209:0:327530:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.474220:0:327530:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.474229:0:327530:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e5d7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327530 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.474259:0:327530:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6d80 x1876350913490432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.474278:0:327530:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431514.474535:0:327530:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431514.474556:0:327530:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431514.474575:0:327530:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c15939a0 00010000:00010000:2.0:1789431514.474582:0:327530:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431514.474593:0:327530:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431514.490219:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431514.490250:0:327532:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431514.490264:0:327532:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431514.490297:0:327532:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431514.490903:0:327532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431514.491067:0:327532:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431514.491082:0:327532:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431514.491095:0:327532:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431514.491100:0:327532:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e5bb 00000080:00010000:3.0:1789431514.491113:0:327532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c160bcb8 00010000:00010000:3.0:1789431514.491118:0:327532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431514.491132:0:327532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431514.493216:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493238:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.493251:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493255:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.493263:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493266:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.493275:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493278:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.493288:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493291:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.493301:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493304:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.493312:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493315:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.493322:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493325:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.493335:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.493356:0:327532:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c3a9cf00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431514.493366:0:327532:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431514.493407:0:327532:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431514.493508:0:327532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431514.493521:0:327532:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431514.493538:0:327532:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431514.493557:0:327532:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7100 x1876350913490688/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431514.493569:0:327532:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431514.493602:0:327532:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327532:1876350913490688:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431514.494129:0:327532:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.500171:0:327532:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327532:1876350913490688:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.500212:0:327532:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500232:0:327532:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500248:0:327532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500258:0:327532:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500272:0:327532:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500283:0:327532:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.500290:0:327532:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500320:0:327532:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500330:0:327532:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500345:0:327532:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.500353:0:327532:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.500360:0:327532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118400/0x1ffb2ee1bd27e5de lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.500383:0:327532:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913490688/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.500400:0:327532:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.500443:0:327532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.500463:0:327532:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431514.500478:0:327532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c160b9a0 00010000:00010000:1.0:1789431514.500486:0:327532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.500498:0:327532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.501155:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501176:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.501200:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501208:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501232:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501239:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501255:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501261:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501279:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501284:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501300:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501307:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501321:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501324:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501334:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501352:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501362:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501366:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.501581:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501587:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.501598:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501602:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501612:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501615:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501623:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501625:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501632:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501635:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.501643:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501646:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501654:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501656:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501665:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501667:0:327532:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.501676:0:327532:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.501690:0:327532:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62f00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.501697:0:327532:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.501729:0:327532:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.501806:0:327532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.501816:0:327532:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.501829:0:327532:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.501896:0:327532:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913490944/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.501908:0:327532:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.501935:0:327532:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327532:1876350913490944:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.502214:0:327532:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.510577:0:327532:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327532:1876350913490944:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.510616:0:327532:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510637:0:327532:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510647:0:327532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510657:0:327532:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510671:0:327532:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510680:0:327532:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.510685:0:327532:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510694:0:327532:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510701:0:327532:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510709:0:327532:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.510717:0:327532:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.510724:0:327532:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5e5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327532 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.510747:0:327532:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913490944/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.510766:0:327532:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.510805:0:327532:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.510820:0:327532:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00010000:1.0:1789431514.510830:0:327532:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c160b9a0 00010000:00010000:1.0:1789431514.510869:0:327532:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.510879:0:327532:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431514.525237:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.525270:0:327535:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.525282:0:327535:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.525345:0:327535:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.525372:0:327535:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431514.525392:0:327535:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431514.525404:0:327535:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431514.525415:0:327535:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.525419:0:327535:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e5bb 00000080:00010000:2.0:1789431514.525429:0:327535:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5bb from it ffffbc42c163fcb8 00010000:00010000:2.0:1789431514.525433:0:327535:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431514.525443:0:327535:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431514.527754:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527781:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.527808:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527815:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.527880:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527887:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.527907:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527914:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.527932:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527937:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.527956:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527962:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.527979:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.527984:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.527999:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.528004:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.528018:0:327535:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.528024:0:327535:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.534586:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534613:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.534638:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534645:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.534666:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534673:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.534686:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534690:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.534699:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534702:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.534708:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534712:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.534719:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534721:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.534729:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534732:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.534741:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.534745:0:327534:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.536115:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.536153:0:327534:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:0.0:1789431514.536157:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.536162:0:327534:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.536167:0:327534:0:(namei.c:2165:do_mkdir()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:0.0:1789431514.536233:0:327534:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431514.536268:0:327534:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431514.536460:0:327534:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431514.536469:0:327534:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431514.536473:0:327534:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62000/0x1ffb2ee1bd27e5bb lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac50d expref: -99 pid: 327530 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000020:00100000:0.0:1789431514.536495:0:327534:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431514.536512:0:327534:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4700 x1876350913491200/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 592/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431514.536542:0:327534:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327534:1876350913491200:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431514.536741:0:327534:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.561238:0:327534:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327534:1876350913491200:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431514.561309:0:327534:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x696:0x0](ffff9811c48721c8) 00000080:00200000:0.0:1789431514.561385:0:327534:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x696:0x0], updating i_size 4096 00000080:00200000:1.0:1789431514.572256:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.572283:0:327537:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.572292:0:327537:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.572330:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.572356:0:327537:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431514.572427:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.572442:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.572454:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.572485:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491328:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431514.572603:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.578487:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491328:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431514.578525:0:327537:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578541:0:327537:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578554:0:327537:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578567:0:327537:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578580:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578589:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578602:0:327537:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.578614:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bde680 x1876350913491328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431514.578629:0:327537:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578645:0:327537:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431514.578653:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431514.578689:0:327537:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431514.578695:0:327537:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e5ec 00000080:00010000:1.0:1789431514.578702:0:327537:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.578706:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431514.578710:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.578719:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431514.587522:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.587554:0:327537:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.587563:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.587605:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.587722:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.587737:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.587753:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.587795:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491456:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.588003:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.593607:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491456:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.593643:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.593664:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.593675:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594272:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594283:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594291:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.594296:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594394:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594405:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594412:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.594421:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594429:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e5f3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.594453:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913491456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.594470:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.594507:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.594522:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431514.594532:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b970 00010000:00010000:1.0:1789431514.594536:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.594545:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431514.594639:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.594656:0:327537:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.594661:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.594682:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.594986:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.594996:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.595007:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.595034:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491584:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.595207:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.601197:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491584:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.601228:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601245:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601253:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601261:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601270:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601276:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.601283:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601291:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601308:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601315:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.601323:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601330:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e5fa lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.601358:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913491584/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.601376:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.601409:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.601424:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431514.601434:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b970 00010000:00010000:1.0:1789431514.601437:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.601447:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.601546:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.601556:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.601566:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.601569:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.601576:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.601579:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.601587:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.601597:0:327537:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.601601:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.601621:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.601666:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601674:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.601684:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.601705:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491712:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.601803:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.604895:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491712:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.604925:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604940:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604953:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604961:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604972:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604980:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ee00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.604985:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604991:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.604998:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.605006:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.605013:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.605019:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e601 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.605041:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913491712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.605059:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.605101:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.605117:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431514.605128:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b970 00010000:00010000:1.0:1789431514.605131:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.605140:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.605236:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.605248:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.605263:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.605267:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.605277:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.605293:0:327537:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388f080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.605300:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.605325:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.605394:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.605402:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.605413:0:327537:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.605421:0:327537:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913491968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.605436:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.605460:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491968:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.605704:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.611235:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913491968:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.611266:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611281:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611289:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611297:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611389:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611396:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.611402:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611410:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611418:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611426:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.611434:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.611441:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e608 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.611467:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913491968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.611484:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.611520:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.611533:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431514.611542:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b9a0 00010000:00010000:1.0:1789431514.611546:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.611553:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.612421:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612442:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.612463:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612470:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.612487:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612495:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.612512:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612518:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.612536:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612542:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.612559:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612565:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.612583:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612588:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.612604:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612609:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.612625:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.612645:0:327537:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388f980, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.612656:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.613116:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.613208:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.613220:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.613234:0:327537:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.613248:0:327537:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913492224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.613259:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.613287:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913492224:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.613777:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.619784:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913492224:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.619823:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619905:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619917:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619929:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619943:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619952:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.619960:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619971:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619981:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.619990:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.620000:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.620010:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e60f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.620043:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913492224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.620060:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.620096:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.620112:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431514.620124:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b9a0 00010000:00010000:1.0:1789431514.620130:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.620142:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.621285:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621324:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.621353:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621357:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.621368:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621371:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.621380:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621383:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.621397:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621402:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.621416:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621420:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.621430:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621433:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.621440:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621443:0:327537:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.621450:0:327537:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.621464:0:327537:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62780, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.621471:0:327537:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.621501:0:327537:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.621587:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.621601:0:327537:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.621618:0:327537:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.622227:0:327537:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913492480/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.622240:0:327537:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.622271:0:327537:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913492480:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.623003:0:327537:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.630261:0:327537:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327537:1876350913492480:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.630297:0:327537:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630404:0:327537:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630419:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630426:0:327537:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630436:0:327537:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630443:0:327537:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ee00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.630449:0:327537:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630460:0:327537:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630471:0:327537:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630481:0:327537:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.630491:0:327537:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.630501:0:327537:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e616 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327537 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.630532:0:327537:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913492480/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.630553:0:327537:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.630598:0:327537:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.630616:0:327537:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431514.630630:0:327537:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154b9a0 00010000:00010000:1.0:1789431514.630635:0:327537:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.630646:0:327537:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431514.642055:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642083:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431514.642108:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642114:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.642134:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642139:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.642156:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642161:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.642174:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642179:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431514.642193:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642199:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.642213:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642217:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.642231:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642236:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431514.642251:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.642257:0:327539:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431514.642917:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.642931:0:327540:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.642938:0:327540:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.642971:0:327540:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.642997:0:327540:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431514.643018:0:327540:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431514.643036:0:327540:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431514.643046:0:327540:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.643051:0:327540:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e5ec 00000080:00010000:2.0:1789431514.643061:0:327540:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e5ec from it ffffbc42c154bcb8 00010000:00010000:2.0:1789431514.643067:0:327540:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431514.643079:0:327540:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431514.644780:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644802:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.644823:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644869:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.644890:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644896:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.644911:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644915:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.644931:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644936:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.644949:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644955:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.644970:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644975:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.644991:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.644998:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.645014:0:327540:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.645020:0:327540:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431514.651111:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.651148:0:327539:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:0.0:1789431514.651153:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.651159:0:327539:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431514.651164:0:327539:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:0.0:1789431514.651227:0:327539:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431514.651262:0:327539:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431514.651383:0:327539:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431514.651390:0:327539:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431514.651394:0:327539:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e000/0x1ffb2ee1bd27e5ec lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac5ae expref: -99 pid: 327537 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:0.0:1789431514.651422:0:327539:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431514.651438:0:327539:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5f80 x1876350913492736/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431514.651466:0:327539:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327539:1876350913492736:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431514.651592:0:327539:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.656956:0:327539:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327539:1876350913492736:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431514.666272:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.666300:0:327542:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.666312:0:327542:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.666353:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.666372:0:327542:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431514.666443:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.666454:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.666466:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.666500:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913492864:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431514.666636:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.671386:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913492864:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431514.671428:0:327542:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671443:0:327542:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671460:0:327542:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671476:0:327542:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671493:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671504:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671513:0:327542:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.671529:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bde680 x1876350913492864/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431514.671549:0:327542:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671569:0:327542:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431514.671579:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431514.671625:0:327542:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431514.671635:0:327542:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e61d 00000080:00010000:1.0:1789431514.671643:0:327542:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.671648:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174bcb8 00010000:00010000:1.0:1789431514.671652:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.671664:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431514.683252:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.683286:0:327542:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.683294:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.684551:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.684635:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.684648:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.684659:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.685252:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913492992:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.685406:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.691402:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913492992:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.691438:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691461:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691468:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691476:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691486:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691494:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.691499:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691506:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691514:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691521:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.691529:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.691536:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e624 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.691557:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913492992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.691579:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.691615:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.691632:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431514.691645:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b970 00010000:00010000:1.0:1789431514.691651:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.691661:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431514.692135:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.692155:0:327542:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.692160:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.692184:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.692240:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.692251:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.692263:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.692291:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493120:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.692552:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.695973:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493120:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.696003:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696018:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696024:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696031:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696041:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696048:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.696053:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696061:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696068:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696077:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.696083:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696094:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e62b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.696118:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913493120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.696136:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.696173:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.696187:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431514.696196:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b970 00010000:00010000:1.0:1789431514.696200:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.696209:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.696316:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.696328:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.696357:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.696361:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.696371:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.696375:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.696383:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.696395:0:327542:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.696400:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.696422:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.696470:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696478:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.696488:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.696513:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493248:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.696617:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.700781:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493248:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.700811:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700827:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700869:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700876:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700886:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700893:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.700897:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700906:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700914:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700921:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.700928:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.700934:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e632 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.700957:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913493248/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.700975:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.701012:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.701026:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431514.701036:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b970 00010000:00010000:1.0:1789431514.701040:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.701049:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.701149:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.701159:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.701172:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.701176:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.701184:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.701197:0:327542:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64d80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.701204:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.701227:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.701289:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.701375:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.701394:0:327542:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.701404:0:327542:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913493504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.701415:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.701441:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493504:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.701698:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.706567:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493504:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.706597:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706613:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706621:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706629:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706638:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706644:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.706649:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706657:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706665:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706671:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.706678:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.706684:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e639 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.706705:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913493504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.706722:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.706759:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.706775:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431514.706798:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b9a0 00010000:00010000:1.0:1789431514.706801:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.706811:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.707368:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707382:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.707396:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707400:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.707410:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707414:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.707423:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707425:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.707433:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707436:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.707444:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707448:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.707456:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707459:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.707470:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707473:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.707483:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.707496:0:327542:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388f680, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.707504:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.707531:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.707606:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.707618:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.707633:0:327542:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.707645:0:327542:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913493760/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.707658:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.707689:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493760:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.708678:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.714901:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913493760:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.714940:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.714959:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.714969:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.714978:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.714998:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.715008:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.715016:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.715025:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.715035:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.715045:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.715054:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.715062:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e640 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.715092:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913493760/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.715110:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.715153:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.715170:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431514.715182:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b9a0 00010000:00010000:1.0:1789431514.715189:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.715200:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.716545:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716569:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.716590:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716597:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.716614:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716620:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.716634:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716639:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.716654:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716660:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.716673:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.716679:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.717073:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.717084:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.717106:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.717113:0:327542:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.717139:0:327542:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.717160:0:327542:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388fc80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.717174:0:327542:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.717216:0:327542:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.717394:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.717410:0:327542:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.717428:0:327542:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.717443:0:327542:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913494016/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.717456:0:327542:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.717494:0:327542:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913494016:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.718408:0:327542:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.721439:0:327542:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327542:1876350913494016:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.721479:0:327542:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721500:0:327542:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721511:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721521:0:327542:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721536:0:327542:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721546:0:327542:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.721554:0:327542:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721569:0:327542:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721580:0:327542:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721590:0:327542:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.721600:0:327542:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.721609:0:327542:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e647 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327542 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.721638:0:327542:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde680 x1876350913494016/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.721658:0:327542:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.721700:0:327542:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.721715:0:327542:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431514.721726:0:327542:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c174b9a0 00010000:00010000:1.0:1789431514.721730:0:327542:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.721739:0:327542:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431514.735997:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.736025:0:327545:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.736034:0:327545:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.736064:0:327545:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.736089:0:327545:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431514.736112:0:327545:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431514.736123:0:327545:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431514.736134:0:327545:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431514.736140:0:327545:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e61d 00000080:00010000:2.0:1789431514.736151:0:327545:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e61d from it ffffbc42c0ddbcb8 00010000:00010000:2.0:1789431514.736157:0:327545:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431514.736169:0:327545:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431514.738246:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.738268:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431514.738283:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.738287:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.738295:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.738298:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.738307:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.738311:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.738321:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.738326:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431514.739038:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.739055:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.739098:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.739105:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.739125:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.739131:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431514.739150:0:327545:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431514.739158:0:327545:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.744224:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744245:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.744257:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744261:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.744269:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744272:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.744282:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744285:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.744293:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744296:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.744304:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744308:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.744316:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744319:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.744327:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744330:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.744349:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.744354:0:327544:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431514.745284:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.745316:0:327544:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:1.0:1789431514.745322:0:327544:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431514.745369:0:327544:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431514.745461:0:327544:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327544 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.745473:0:327544:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327544 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.745489:0:327544:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327544 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.745524:0:327544:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327544:1876350913494144:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:1.0:1789431514.745775:0:327544:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.751599:0:327544:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327544:1876350913494144:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431514.751642:0:327544:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751677:0:327544:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x696:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751690:0:327544:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751707:0:327544:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751725:0:327544:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751744:0:327544:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751758:0:327544:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751807:0:327544:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.751828:0:327544:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bde680 x1876350913494144/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431514.751920:0:327544:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.751936:0:327544:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1686 (0 0) 00000080:00200000:1.0:1789431514.751947:0:327544:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431514.751991:0:327544:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x696:0x0], updating i_size 4096 00000080:00200000:1.0:1789431514.751998:0:327544:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x696:0x0](ffff9811c48721c8): rc = 0 00000080:00010000:1.0:1789431514.752006:0:327544:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x696:0x0](ffff9811c48721c8) for lock 0x1ffb2ee1bd27e64e 00000080:00010000:1.0:1789431514.752040:0:327544:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x696:0x0](ffff9811c48721c8) 00000080:00010000:1.0:1789431514.752046:0:327544:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e64e from it ffffbc42c1ed3e08 00010000:00010000:1.0:1789431514.752051:0:327544:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.752065:0:327544:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431514.752117:0:327544:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431514.752131:0:327544:0:(namei.c:2254:ll_rmdir()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431514.752179:0:327544:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431514.752214:0:327544:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431514.752285:0:327544:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.752381:0:327544:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.752394:0:327544:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431514.752402:0:327544:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431514.752408:0:327544:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e64e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x696:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac6d4 expref: -99 pid: 327544 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.752426:0:327544:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e61d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac63a expref: -99 pid: 327542 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431514.752445:0:327544:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431514.752456:0:327544:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde680 x1876350913494400/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431514.752492:0:327544:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327544:1876350913494400:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431514.752631:0:327544:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.764247:0:327544:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327544:1876350913494400:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431514.764355:0:327544:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x696:0x0](ffff9811c48721c8) 00000080:00200000:1.0:1789431514.781088:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.781121:0:327547:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.781135:0:327547:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.781172:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.781195:0:327547:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431514.781273:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.781286:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.781302:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.784247:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494528:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431514.784532:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.789585:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494528:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431514.789620:0:327547:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789635:0:327547:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789649:0:327547:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789660:0:327547:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789693:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789703:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789712:0:327547:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.789725:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdce00 x1876350913494528/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431514.789740:0:327547:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789754:0:327547:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431514.789761:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431514.789803:0:327547:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431514.789810:0:327547:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e655 00000080:00010000:1.0:1789431514.789816:0:327547:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.789820:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfbcb8 00010000:00010000:1.0:1789431514.789824:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.789880:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431514.792415:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1789431514.795050:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431514.795080:0:327547:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431514.795086:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431514.795127:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431514.795206:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.795219:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.795230:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.795263:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494656:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431514.795438:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c84fd540 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.804631:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494656:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431514.804663:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804686:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804694:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804702:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804713:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804720:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.804725:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804733:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804741:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804750:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.804757:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.804765:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e65c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.804786:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd5f80 x1876350913494656/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.804810:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.805146:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.805165:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431514.805176:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb970 00010000:00010000:1.0:1789431514.805181:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.805201:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431514.805314:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.805344:0:327547:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.805350:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.805372:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.805424:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.805432:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.805443:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.805471:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494784:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.805596:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.811661:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494784:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.812447:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812473:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812483:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812493:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812507:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812516:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.812525:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812536:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812549:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812558:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.812569:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.812580:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e663 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.813996:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913494784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.814018:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.814067:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.814087:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431514.814099:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb970 00010000:00010000:1.0:1789431514.814107:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.814122:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.814240:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.814251:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.814264:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.814269:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.814285:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.814291:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.814308:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.814523:0:327547:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.814531:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.814560:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.814666:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.814676:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.814747:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.814929:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494912:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.815046:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.821731:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913494912:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.821779:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821800:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821808:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821816:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821872:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821879:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.821885:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821892:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821901:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821909:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.821917:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.821924:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e66a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.821945:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913494912/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.821961:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.821997:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.822012:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431514.822022:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb970 00010000:00010000:1.0:1789431514.822026:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.822035:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.822130:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.822142:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.822161:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.822164:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.822173:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.822187:0:327547:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64a80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.822194:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.822228:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.822285:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.822293:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.822310:0:327547:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.822393:0:327547:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913495168/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.822404:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.822430:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495168:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.822695:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.833246:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495168:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.833288:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833309:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833345:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833354:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833366:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833372:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.833378:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833385:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833394:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833401:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.833409:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.833416:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e671 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.833440:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913495168/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.833459:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.833498:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.833514:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431514.833528:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb9a0 00010000:00010000:1.0:1789431514.833534:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.833547:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.834310:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834330:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.834359:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834365:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.834374:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834376:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.834383:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834386:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.834394:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834397:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.834405:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834409:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.834418:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834421:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.834428:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834431:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.834438:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.834451:0:327547:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64300, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.834458:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.834487:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.834562:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.834572:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.834585:0:327547:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.834597:0:327547:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913495424/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.834608:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.834651:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495424:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.834945:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.841178:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495424:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.841212:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841231:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841239:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841248:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841258:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841271:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.841277:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841287:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841306:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841314:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.841323:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.841330:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e678 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.841364:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913495424/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.841381:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.841420:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.841436:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431514.841447:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb9a0 00010000:00010000:1.0:1789431514.841451:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.841465:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.842063:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842080:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.842093:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842097:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.842105:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842108:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.842116:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842118:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.842125:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842127:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.842135:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842139:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.842146:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842148:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.842157:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842160:0:327547:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.842168:0:327547:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.842183:0:327547:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65c80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.842190:0:327547:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.842220:0:327547:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.842297:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.842309:0:327547:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.842325:0:327547:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.842345:0:327547:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913495680/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.842356:0:327547:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.842383:0:327547:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495680:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.844432:0:327547:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.853467:0:327547:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327547:1876350913495680:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.853502:0:327547:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853521:0:327547:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853530:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853540:0:327547:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853553:0:327547:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853561:0:327547:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.853567:0:327547:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853580:0:327547:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853587:0:327547:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853595:0:327547:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.853601:0:327547:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.853607:0:327547:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e67f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327547 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.853630:0:327547:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913495680/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.853646:0:327547:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.853686:0:327547:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.853701:0:327547:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431514.853712:0:327547:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1bfb9a0 00010000:00010000:1.0:1789431514.853717:0:327547:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431514.853726:0:327547:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.878044:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431514.878074:0:327550:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431514.878085:0:327550:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431514.878115:0:327550:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431514.878142:0:327550:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.878161:0:327550:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431514.878170:0:327550:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431514.878181:0:327550:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431514.878186:0:327550:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e655 00000080:00010000:1.0:1789431514.878194:0:327550:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e655 from it ffffbc42c1c13cb8 00010000:00010000:1.0:1789431514.878199:0:327550:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431514.878209:0:327550:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431514.881557:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881581:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.881599:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881604:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.881615:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881618:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.881629:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881631:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.881639:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881642:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.881649:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881653:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.881662:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881665:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.881675:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881678:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.881688:0:327550:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.881692:0:327550:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431514.885279:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885581:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431514.885602:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885609:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.885619:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885623:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.885632:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885636:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.885646:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885649:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431514.885659:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885662:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.885671:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885673:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.885681:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885684:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431514.885694:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.885698:0:327549:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431514.886761:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431514.887203:0:327549:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:3.0:1789431514.887209:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431514.887216:0:327549:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431514.887221:0:327549:0:(namei.c:2010:ll_mknod()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) mode 10644 dev 0 00010000:00010000:3.0:1789431514.887282:0:327549:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431514.887405:0:327549:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431514.887508:0:327549:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431514.887516:0:327549:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431514.887520:0:327549:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27e655 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac6f7 expref: -99 pid: 327547 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1789431514.887543:0:327549:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431514.887559:0:327549:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5500 x1876350913495936/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 544/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431514.887588:0:327549:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327549:1876350913495936:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431514.887721:0:327549:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.892544:0:327549:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327549:1876350913495936:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431514.892600:0:327549:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x698:0x0](ffff9811c48742c8) 00000080:00200000:0.0:1789431514.892607:0:327549:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x698:0x0], updating i_size 0 00000080:00200000:2.0:1789431514.902874:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431514.902908:0:327552:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431514.902922:0:327552:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431514.902960:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431514.902986:0:327552:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431514.903075:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.903090:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.903105:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431514.903149:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496064:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431514.903562:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431514.916758:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496064:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431514.916795:0:327552:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916815:0:327552:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916847:0:327552:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916860:0:327552:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916875:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916884:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916892:0:327552:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431514.916904:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2102a00 x1876350913496064/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431514.916919:0:327552:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431514.916934:0:327552:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431514.916941:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431514.918243:0:327552:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431514.918261:0:327552:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e686 00000080:00010000:3.0:1789431514.918272:0:327552:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431514.918279:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1bcb8 00010000:00010000:3.0:1789431514.918288:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431514.918306:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431514.927743:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.927781:0:327552:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.927788:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.927985:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.928131:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.928144:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.928156:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.928187:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496192:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.928458:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.935876:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496192:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.935916:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.935941:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.935951:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.935962:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.935976:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.935986:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.935992:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936023:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936033:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936041:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.936048:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936054:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e68d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.936077:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913496192/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.936100:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.936146:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.936164:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431514.936179:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b970 00010000:00010000:1.0:1789431514.936186:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431514.936197:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431514.936300:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.936542:0:327552:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.936552:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.936585:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.936661:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936674:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.936688:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.936724:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496320:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.936909:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.942633:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496320:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.942666:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942694:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942703:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942710:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942719:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942726:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.942732:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942739:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942756:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942763:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.942771:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.942795:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e694 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.942817:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913496320/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.942869:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.942907:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.942922:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431514.942931:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b970 00010000:00010000:1.0:1789431514.942935:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431514.942942:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431514.943475:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.943491:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.943508:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.943511:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.943519:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.943522:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.943533:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.943546:0:327552:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431514.943551:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431514.943574:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431514.943626:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.943635:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.943644:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.943668:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496448:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.944184:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.949629:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496448:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.949659:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949675:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949694:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949702:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949713:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949720:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.949725:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949734:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949742:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949758:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.949766:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.949773:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e69b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.949803:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913496448/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.949821:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.949893:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.949909:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431514.949920:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b970 00010000:00010000:1.0:1789431514.949924:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431514.949934:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431514.950069:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.950081:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.950096:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.950099:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.950108:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.950122:0:327552:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64780, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.950129:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.950152:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.950210:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.950218:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.950231:0:327552:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.950239:0:327552:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913496704/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.950251:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.950281:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496704:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.950565:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.961502:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496704:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.961534:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961551:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961559:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961567:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961577:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961584:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.961589:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961597:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961606:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961612:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.961619:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.961624:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6a2 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.961644:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913496704/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.961661:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.962423:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.962444:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431514.962455:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b9a0 00010000:00010000:1.0:1789431514.962460:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431514.962471:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431514.963113:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963127:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.963139:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963143:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.963150:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963153:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.963160:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963163:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.963170:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963172:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.963179:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963183:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.963190:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963192:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.963198:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963201:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.963208:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.963220:0:327552:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64600, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431514.963227:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431514.963250:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431514.963600:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.963611:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431514.963625:0:327552:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431514.963639:0:327552:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913496960/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431514.963648:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431514.963676:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496960:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431514.964232:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431514.969228:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913496960:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431514.969269:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969285:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969295:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969303:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969317:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969324:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431514.969330:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969352:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969360:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969367:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431514.969375:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431514.969381:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6a9 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431514.969408:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913496960/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431514.969426:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431514.969466:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.969481:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431514.969491:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b9a0 00010000:00010000:1.0:1789431514.969495:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431514.969503:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431514.970693:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970705:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431514.970712:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970715:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.970719:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970721:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.970726:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970728:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.970732:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970734:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431514.970738:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970740:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.970745:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970747:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.970755:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970758:0:327552:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431514.970765:0:327552:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431514.970888:0:327552:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65680, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431514.972153:0:327552:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431514.972202:0:327552:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431514.972345:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.972359:0:327552:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431514.972377:0:327552:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431514.972395:0:327552:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7100 x1876350913497216/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431514.972409:0:327552:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431514.972438:0:327552:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913497216:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431514.972742:0:327552:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431514.979427:0:327552:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327552:1876350913497216:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431514.979463:0:327552:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979479:0:327552:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979494:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979502:0:327552:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979512:0:327552:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979519:0:327552:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431514.979524:0:327552:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979532:0:327552:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979539:0:327552:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979549:0:327552:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431514.979558:0:327552:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431514.979568:0:327552:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e6b0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327552 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431514.979601:0:327552:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7100 x1876350913497216/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431530 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431514.979641:0:327552:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431514.979692:0:327552:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431514.979712:0:327552:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431514.979726:0:327552:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1c1b9a0 00010000:00010000:0.0:1789431514.979733:0:327552:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431514.979744:0:327552:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431514.991105:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431514.991133:0:327555:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431514.991141:0:327555:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431514.991169:0:327555:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431514.991191:0:327555:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431514.991209:0:327555:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431514.991218:0:327555:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431514.991229:0:327555:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431514.991233:0:327555:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e686 00000080:00010000:3.0:1789431514.991242:0:327555:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e686 from it ffffbc42c1ddbcb8 00010000:00010000:3.0:1789431514.991246:0:327555:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431514.991256:0:327555:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431515.004584:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004606:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431515.004634:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004639:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.004649:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004652:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.004663:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004667:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.004686:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004690:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.004702:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004706:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.004715:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004718:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.004726:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004729:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.004736:0:327555:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.004740:0:327555:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.017134:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017156:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.017171:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017176:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.017186:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017190:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.017199:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017202:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.017210:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017213:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.017222:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017226:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.017235:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017238:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.017248:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017251:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.017261:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.017264:0:327554:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.023660:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.023764:0:327554:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:0.0:1789431515.023770:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431515.023776:0:327554:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431515.023781:0:327554:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:0.0:1789431515.024194:0:327554:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.024237:0:327554:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431515.024545:0:327554:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.024554:0:327554:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431515.024557:0:327554:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e686 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac791 expref: -99 pid: 327552 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:0.0:1789431515.024581:0:327554:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431515.024598:0:327554:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7100 x1876350913497472/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431515.024635:0:327554:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327554:1876350913497472:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431515.025471:0:327554:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.029306:0:327554:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327554:1876350913497472:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431515.038628:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431515.038665:0:327557:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431515.038675:0:327557:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431515.038707:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431515.038727:0:327557:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431515.038801:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.038816:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.038863:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.038914:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497600:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431515.041168:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.050954:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497600:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431515.050995:0:327557:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051009:0:327557:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051024:0:327557:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051037:0:327557:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051051:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051060:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051068:0:327557:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431515.051081:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc6a00 x1876350913497600/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431515.051257:0:327557:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051273:0:327557:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431515.051280:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431515.051322:0:327557:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431515.051330:0:327557:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e6b7 00000080:00010000:3.0:1789431515.051347:0:327557:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431515.051351:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c43cb8 00010000:00010000:3.0:1789431515.051355:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.051365:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431515.054484:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.054509:0:327557:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431515.054516:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431515.054549:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431515.054633:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.054645:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.054657:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.054694:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497728:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431515.055008:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.059214:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497728:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431515.059247:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059262:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059271:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059279:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059289:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059440:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431515.059446:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059456:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059465:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059473:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431515.059482:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.059489:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e6be lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431515.059514:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6a00 x1876350913497728/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431515.059531:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431515.059567:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431515.059581:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431515.059591:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c43970 00010000:00010000:3.0:1789431515.059595:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431515.059604:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431515.059965:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.059987:0:327557:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431515.059992:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.060168:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.060216:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.060223:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.060231:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.060250:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497856:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.060452:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.064239:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497856:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.064272:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064290:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064299:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064307:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064326:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064342:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.064348:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064356:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064364:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064372:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.064380:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.064387:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e6c5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.064408:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2102d80 x1876350913497856/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.064435:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.064470:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.064485:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431515.064495:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c43970 00010000:00010000:1.0:1789431515.064499:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.064508:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431515.064601:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.064899:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.064919:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.064923:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.064933:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.064936:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.064945:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.064963:0:327557:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.064968:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.064992:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.065123:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.065132:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.065143:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.065173:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497984:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.065282:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.069666:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913497984:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.069705:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069722:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069729:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069737:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069746:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069753:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.069758:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069765:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069774:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069790:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.069800:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.069806:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6cc lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.069826:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913497984/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.069891:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.069930:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.069946:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431515.069957:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c43970 00010000:00010000:1.0:1789431515.069961:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.069969:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431515.070068:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.070080:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.070094:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.070097:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.070104:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.070121:0:327557:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64000, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.070128:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.070152:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.070214:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.070221:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.070234:0:327557:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.070241:0:327557:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913498240/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.070250:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.070274:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498240:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.071737:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.078949:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498240:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.078985:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079002:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079010:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079018:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079029:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079038:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.079044:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079052:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079059:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079067:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.079074:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079080:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6d3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.079104:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913498240/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.079122:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.079160:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.079176:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431515.079187:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.079191:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.079205:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431515.079684:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079697:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.079710:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079715:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.079724:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079727:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.079735:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079738:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.079748:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079750:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.079758:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079762:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.079773:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079776:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.079785:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079788:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.079796:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.079809:0:327557:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65b00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.079817:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.079906:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.079970:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.079979:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.079996:0:327557:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.080006:0:327557:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913498496/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.080017:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.080042:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498496:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.080281:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.083401:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498496:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.083431:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083444:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083450:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083456:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083464:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083468:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.083472:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083480:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083486:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083490:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.083496:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.083500:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e6da lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.083517:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913498496/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.083531:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.083562:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.083574:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431515.083582:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.083586:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.083598:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.084508:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084521:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.084531:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084535:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.084544:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084546:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.084556:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084558:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.084569:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084572:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.084581:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084585:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.084593:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084596:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.084610:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084613:0:327557:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.084623:0:327557:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.084636:0:327557:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388f080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.084643:0:327557:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.084740:0:327557:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.085016:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.085026:0:327557:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.086235:0:327557:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.086470:0:327557:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913498752/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.086485:0:327557:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.086518:0:327557:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498752:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.087047:0:327557:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.094492:0:327557:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327557:1876350913498752:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.094529:0:327557:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094545:0:327557:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094552:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094561:0:327557:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094571:0:327557:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094578:0:327557:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.094584:0:327557:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094593:0:327557:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094600:0:327557:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094608:0:327557:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.094616:0:327557:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.094622:0:327557:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327557 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.094715:0:327557:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdce00 x1876350913498752/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.094735:0:327557:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.094775:0:327557:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.095139:0:327557:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.095151:0:327557:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.095156:0:327557:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.095165:0:327557:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.105560:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431515.105582:0:327560:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431515.105591:0:327560:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431515.105618:0:327560:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431515.105655:0:327560:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.105828:0:327560:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.105863:0:327560:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431515.105875:0:327560:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431515.105879:0:327560:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e6b7 00000080:00010000:3.0:1789431515.105888:0:327560:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6b7 from it ffffbc42c1677cb8 00010000:00010000:3.0:1789431515.105894:0:327560:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.105904:0:327560:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431515.107970:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.107989:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.108075:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108080:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.108090:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108094:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.108104:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108106:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.108115:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108119:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.108127:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108132:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.108140:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108143:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.108150:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108153:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.108162:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.108165:0:327559:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431515.108530:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108536:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.108545:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108549:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.108558:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108561:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.108569:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108572:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.108579:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108582:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.108590:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108593:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.108601:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108604:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.108612:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108615:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.108621:0:327560:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.108624:0:327560:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431515.109748:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.109776:0:327559:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:1.0:1789431515.109782:0:327559:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431515.109815:0:327559:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431515.110137:0:327559:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327559 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.110150:0:327559:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327559 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.110162:0:327559:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327559 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.110195:0:327559:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327559:1876350913498880:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:1.0:1789431515.110330:0:327559:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.115162:0:327559:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327559:1876350913498880:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431515.115198:0:327559:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115226:0:327559:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x698:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115235:0:327559:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115248:0:327559:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115261:0:327559:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115276:0:327559:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115285:0:327559:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115293:0:327559:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.115388:0:327559:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdce00 x1876350913498880/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431515.115402:0:327559:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115415:0:327559:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1688 (0 0) 00000080:00200000:1.0:1789431515.115422:0:327559:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431515.115449:0:327559:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x698:0x0], updating i_size 0 00000080:00200000:1.0:1789431515.115453:0:327559:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x698:0x0](ffff9811c48742c8): rc = 0 00000080:00010000:1.0:1789431515.115458:0:327559:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x698:0x0](ffff9811c48742c8) for lock 0x1ffb2ee1bd27e6e8 00000080:00010000:1.0:1789431515.115469:0:327559:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x698:0x0](ffff9811c48742c8) 00000080:00010000:1.0:1789431515.115473:0:327559:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6e8 from it ffffbc42c1c43e00 00010000:00010000:1.0:1789431515.115476:0:327559:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.115487:0:327559:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431515.115524:0:327559:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431515.115534:0:327559:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431515.115570:0:327559:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431515.115592:0:327559:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431515.116428:0:327559:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.116493:0:327559:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.116506:0:327559:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431515.116512:0:327559:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431515.116515:0:327559:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6e8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x698:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac8b7 expref: -99 pid: 327559 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.116532:0:327559:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119200/0x1ffb2ee1bd27e6b7 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac81d expref: -99 pid: 327557 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431515.116552:0:327559:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431515.116562:0:327559:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdce00 x1876350913499136/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431515.116593:0:327559:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327559:1876350913499136:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431515.116984:0:327559:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.121227:0:327559:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327559:1876350913499136:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431515.121376:0:327559:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x698:0x0](ffff9811c48742c8) 00000080:00200000:2.0:1789431515.134401:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431515.134429:0:327562:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431515.134440:0:327562:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431515.134469:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431515.134488:0:327562:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431515.134562:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.134573:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.134584:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.134615:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499264:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431515.135400:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.141095:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499264:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431515.141135:0:327562:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141152:0:327562:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141217:0:327562:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141233:0:327562:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141250:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141260:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141267:0:327562:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.141289:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2100700 x1876350913499264/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431515.141305:0:327562:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141321:0:327562:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431515.141329:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431515.141377:0:327562:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431515.141384:0:327562:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e6ef 00000080:00010000:2.0:1789431515.141389:0:327562:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431515.141392:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148bcb8 00010000:00010000:2.0:1789431515.141396:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.141405:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431515.146607:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.146773:0:327562:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.146780:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.146816:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.146925:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.146937:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.146948:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.146978:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499392:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.147365:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.150816:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499392:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.150871:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150888:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150896:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150908:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150918:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150925:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.150930:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150938:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150946:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150954:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.150962:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.150968:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e6f6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.150991:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913499392/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.151018:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.151057:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.151072:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431515.151083:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b970 00010000:00010000:1.0:1789431515.151087:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.151096:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431515.151197:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.151215:0:327562:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.151219:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.151242:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.151291:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.151300:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.151310:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.151346:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499520:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.151449:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.154357:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499520:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.154387:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154403:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154412:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154419:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154429:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154437:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.154442:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154450:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154458:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154466:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.154473:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.154479:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e6fd lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.154500:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913499520/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.154515:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.154549:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.154565:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431515.154575:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b970 00010000:00010000:1.0:1789431515.154578:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.154587:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431515.154699:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.154711:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.154725:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.154729:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.154739:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.154742:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.154751:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.154762:0:327562:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.154767:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.154789:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.155188:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.155199:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.155211:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.155243:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499648:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.155449:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.159156:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499648:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.159197:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159213:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159221:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159229:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159240:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159247:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.159252:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159260:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159267:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159275:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.159283:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.159289:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e704 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.159312:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913499648/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.159329:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.159378:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.159391:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431515.159400:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b970 00010000:00010000:1.0:1789431515.159404:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.159414:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431515.159521:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.159532:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.159548:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.159551:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.159559:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.159574:0:327562:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388e000, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.159583:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.160204:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.160331:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.160352:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.160366:0:327562:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.160377:0:327562:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913499904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.160387:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.160414:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499904:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.161955:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.166554:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913499904:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.166587:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166603:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166611:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166618:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166628:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166636:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.166641:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166650:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166657:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166665:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.166672:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.166679:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e70b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.166703:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913499904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.166719:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.166758:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.166772:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431515.167164:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b9a0 00010000:00010000:1.0:1789431515.167171:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.167187:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431515.169569:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169600:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.169648:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169654:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.169663:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169666:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.169685:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169689:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.169701:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169705:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.169717:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169722:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.169732:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169735:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.169743:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169746:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.169752:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.169767:0:327562:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c388f080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.169775:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.170144:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.170245:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.170257:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.170276:0:327562:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.170305:0:327562:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913500160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.170317:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.170354:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913500160:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.170623:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.174278:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913500160:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.174503:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174518:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174525:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174533:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174542:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174548:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.174552:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174564:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174571:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174577:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.174584:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.174590:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e712 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.174611:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913500160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.174627:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.174662:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.174685:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431515.174696:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b9a0 00010000:00010000:1.0:1789431515.174700:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.174708:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.175206:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175220:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.175547:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175554:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.175573:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175576:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.175585:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175588:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.175599:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175601:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.175608:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175611:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.175616:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175618:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.175624:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175626:0:327562:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.175631:0:327562:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.175641:0:327562:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a3b380, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.175646:0:327562:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.175672:0:327562:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.175745:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.175755:0:327562:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.175769:0:327562:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.175792:0:327562:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913500416/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.175805:0:327562:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.175881:0:327562:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913500416:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.176118:0:327562:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.179615:0:327562:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327562:1876350913500416:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.179641:0:327562:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179654:0:327562:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179659:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179665:0:327562:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179673:0:327562:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179677:0:327562:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.179688:0:327562:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179696:0:327562:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179703:0:327562:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179711:0:327562:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.179719:0:327562:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.179725:0:327562:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e719 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327562 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.179748:0:327562:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913500416/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.179771:0:327562:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.179816:0:327562:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.179961:0:327562:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.179974:0:327562:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c148b9a0 00010000:00010000:1.0:1789431515.179978:0:327562:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.179987:0:327562:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431515.193165:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431515.193191:0:327565:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431515.193200:0:327565:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431515.193230:0:327565:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431515.193265:0:327565:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431515.193284:0:327565:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431515.193303:0:327565:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431515.193312:0:327565:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431515.193316:0:327565:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e6ef 00000080:00010000:2.0:1789431515.193324:0:327565:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e6ef from it ffffbc42c1bebcb8 00010000:00010000:2.0:1789431515.193328:0:327565:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431515.193354:0:327565:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431515.194819:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194865:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.194882:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194886:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.194895:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194899:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.194908:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194912:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.194921:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194924:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.194933:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194936:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.194945:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194948:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.194956:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194959:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.194967:0:327565:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.194971:0:327565:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.195170:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195177:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.195186:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195191:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.195200:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195203:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.195212:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195215:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.195224:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195227:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.195236:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195239:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.195248:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195251:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.195260:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195263:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.195271:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.195274:0:327564:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.196014:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.196113:0:327564:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:0.0:1789431515.196119:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431515.196126:0:327564:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431515.196133:0:327564:0:(namei.c:2010:ll_mknod()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) mode 60644 dev 100002 00010000:00010000:0.0:1789431515.196192:0:327564:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.196227:0:327564:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431515.196323:0:327564:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.196343:0:327564:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431515.196347:0:327564:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e6ef lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac8da expref: -99 pid: 327562 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:0.0:1789431515.196371:0:327564:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431515.196458:0:327564:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7100 x1876350913500672/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 544/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431515.196490:0:327564:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327564:1876350913500672:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431515.196617:0:327564:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.212052:0:327564:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327564:1876350913500672:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431515.212138:0:327564:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x69a:0x0](ffff9811c4871988) 00000080:00200000:0.0:1789431515.212146:0:327564:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69a:0x0], updating i_size 0 00000080:00200000:1.0:1789431515.221220:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431515.221249:0:327567:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431515.227202:0:327567:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431515.227252:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431515.227275:0:327567:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431515.227365:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.227377:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.227388:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.227424:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913500800:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431515.227571:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.233825:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913500800:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431515.233895:0:327567:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233910:0:327567:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233924:0:327567:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233939:0:327567:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233954:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233963:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.233973:0:327567:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.233987:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdf480 x1876350913500800/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431515.234004:0:327567:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.234020:0:327567:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.234027:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431515.234063:0:327567:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431515.234071:0:327567:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e720 00000080:00010000:1.0:1789431515.234077:0:327567:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431515.234081:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c53cb8 00010000:00010000:1.0:1789431515.234084:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.234094:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431515.238400:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.238430:0:327567:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.238437:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.238472:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.238553:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.238566:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.238578:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.238612:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913500928:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.238747:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.249812:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913500928:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.249893:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249913:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249927:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249939:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249950:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249958:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.249963:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249974:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249987:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.249995:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.250002:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.250009:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e727 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.250036:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd4a80 x1876350913500928/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.250199:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.250241:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.250257:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431515.250268:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.250272:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.250281:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431515.250388:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.250407:0:327567:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.250412:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.250445:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.250495:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.250503:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.250514:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.250541:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501056:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.250650:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.257058:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501056:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.257090:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257108:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257116:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257123:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257133:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257141:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.257146:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257153:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257160:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257167:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.257176:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257183:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e72e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.257205:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd4a80 x1876350913501056/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.257222:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.257261:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431515.257276:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1789431515.257286:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.257290:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431515.257300:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431515.257412:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.257424:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.257455:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.257459:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.257470:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.257473:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.257481:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.257494:0:327567:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.257500:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.257525:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.257575:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257583:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.257594:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.257618:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501184:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.257722:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.262306:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501184:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.262349:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262366:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262374:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262381:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262392:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262399:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.262405:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262413:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262422:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262440:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.262448:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.262456:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e735 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.262479:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd4a80 x1876350913501184/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.262497:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.262536:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431515.262551:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1789431515.262563:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.262567:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431515.262577:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431515.262688:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.262699:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.262716:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.262719:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.262731:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.262746:0:327567:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8194000, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431515.262754:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431515.262778:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431515.263177:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.263189:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431515.263204:0:327567:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431515.263213:0:327567:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4a80 x1876350913501440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431515.263225:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.263253:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501440:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.263749:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.270513:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501440:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.270543:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270559:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270572:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270580:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270591:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270597:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.270603:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270610:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270617:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270624:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.270631:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.270637:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2000/0x1ffb2ee1bd27e73c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.270659:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd4a80 x1876350913501440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.270676:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.270714:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.270729:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431515.270739:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c539a0 00010000:00010000:1.0:1789431515.270743:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.270751:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431515.272554:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272574:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.272587:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272591:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.272712:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272718:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.272735:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272740:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.272748:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272751:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.272760:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272763:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.272771:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272773:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.272782:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272785:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.272792:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.272808:0:327567:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65500, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.272817:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.273015:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.273108:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.273119:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.273132:0:327567:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.273148:0:327567:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913501696/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.273160:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.273188:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501696:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.273453:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.280828:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501696:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.280903:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280921:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280930:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280939:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280953:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280961:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.280966:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280974:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280982:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.280990:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.280998:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.281006:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e743 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.281030:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913501696/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.281047:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.281097:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.281114:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431515.281125:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c539a0 00010000:00010000:1.0:1789431515.281130:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.281140:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.281808:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281823:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.281868:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281873:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.281883:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281886:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.281896:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281900:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.281909:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281912:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.281922:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281925:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.281934:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281937:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.281945:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281947:0:327567:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.281955:0:327567:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.281967:0:327567:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5424300, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.281974:0:327567:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.281998:0:327567:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.282131:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.282140:0:327567:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.282154:0:327567:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.282164:0:327567:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913501952/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.282176:0:327567:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.282202:0:327567:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501952:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.282543:0:327567:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.287387:0:327567:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327567:1876350913501952:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.287421:0:327567:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287581:0:327567:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287592:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287601:0:327567:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287741:0:327567:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287751:0:327567:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.287757:0:327567:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287771:0:327567:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287791:0:327567:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287800:0:327567:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.287808:0:327567:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.287817:0:327567:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e74a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327567 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.287869:0:327567:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913501952/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.287888:0:327567:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.287929:0:327567:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.287945:0:327567:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.287956:0:327567:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c1c539a0 00010000:00010000:1.0:1789431515.287961:0:327567:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.287971:0:327567:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431515.299077:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431515.299105:0:327570:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431515.299115:0:327570:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431515.299142:0:327570:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431515.299167:0:327570:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431515.299185:0:327570:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431515.299193:0:327570:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431515.299202:0:327570:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431515.299206:0:327570:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e720 00000080:00010000:1.0:1789431515.299215:0:327570:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e720 from it ffffbc42c148bcb8 00010000:00010000:1.0:1789431515.299220:0:327570:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431515.299230:0:327570:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431515.303696:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.303720:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.304138:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.304146:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.307123:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307145:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.307168:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307171:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.307184:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307187:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.307198:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307201:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.307211:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307214:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.307226:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307242:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.307253:0:327570:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.307258:0:327570:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431515.309726:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309751:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431515.309771:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309776:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.309787:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309790:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.309799:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309803:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.309812:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309815:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.309825:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309829:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.309865:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309870:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.309880:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309883:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.309893:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.309898:0:327569:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431515.313717:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.313756:0:327569:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:3.0:1789431515.313761:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431515.313767:0:327569:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431515.313773:0:327569:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:3.0:1789431515.313858:0:327569:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431515.313893:0:327569:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431515.313997:0:327569:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431515.314005:0:327569:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431515.314008:0:327569:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27e720 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac974 expref: -99 pid: 327567 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1789431515.314031:0:327569:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431515.314047:0:327569:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6a00 x1876350913502208/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431515.314081:0:327569:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327569:1876350913502208:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431515.315998:0:327569:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.320604:0:327569:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327569:1876350913502208:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431515.339436:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431515.339466:0:327572:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431515.339476:0:327572:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431515.339508:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431515.339527:0:327572:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431515.339602:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.339614:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.339627:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.339663:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502336:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431515.339794:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.346563:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502336:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431515.346601:0:327572:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346617:0:327572:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346639:0:327572:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346652:0:327572:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346671:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346680:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346689:0:327572:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.346702:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc5880 x1876350913502336/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431515.346717:0:327572:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346733:0:327572:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431515.346741:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431515.346779:0:327572:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431515.346786:0:327572:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e751 00000080:00010000:2.0:1789431515.346793:0:327572:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431515.346797:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148bcb8 00010000:00010000:2.0:1789431515.346801:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.346810:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431515.351377:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.351409:0:327572:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.351415:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.351451:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.351545:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.351558:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.351570:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.351606:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502464:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.353660:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.357026:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502464:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.357055:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357070:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357077:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357084:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357092:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357099:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.357104:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357110:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357117:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357123:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.357129:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.357135:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e758 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.357153:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913502464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.357167:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.357196:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.357208:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431515.357229:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b970 00010000:00010000:2.0:1789431515.357233:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.357241:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431515.357729:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.357744:0:327572:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.357748:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.357765:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.358008:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.358017:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.358027:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.358049:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502592:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.358143:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.364887:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502592:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.364910:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364923:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364931:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364939:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364947:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364954:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.364959:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364966:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364973:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364979:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.364985:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.364991:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62400/0x1ffb2ee1bd27e75f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.365008:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913502592/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.365024:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.365053:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.365067:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431515.365076:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b970 00010000:00010000:2.0:1789431515.365080:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.365089:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431515.365160:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.365168:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.365177:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.365181:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.365188:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.365191:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.365198:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.365207:0:327572:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.365211:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.365228:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.365266:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.365273:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.365424:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.365452:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502720:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.365533:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.373015:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502720:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.373049:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373067:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373075:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373082:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373093:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373102:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.373108:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373120:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373129:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373139:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.373148:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.373155:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e766 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.373177:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913502720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.373194:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431515.373607:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431515.373775:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431515.373788:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b970 00010000:00010000:3.0:1789431515.373793:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431515.373803:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431515.374046:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.374058:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.374077:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.374081:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.374091:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.374107:0:327572:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c121b680, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431515.374116:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431515.374144:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431515.374211:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.374220:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431515.374233:0:327572:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431515.374244:0:327572:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4000 x1876350913502976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431515.374254:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.374280:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502976:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431515.375069:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.380291:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913502976:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.380405:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380426:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380436:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380445:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380457:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380465:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.380471:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380479:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380488:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380497:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.380505:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.380512:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119a00/0x1ffb2ee1bd27e76d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.380536:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4000 x1876350913502976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.380554:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.380595:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.380611:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:0.0:1789431515.380621:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b9a0 00010000:00010000:0.0:1789431515.380625:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.380634:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431515.381168:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381184:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.381197:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381202:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.381210:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381213:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.381222:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381226:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.381235:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381238:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.381246:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381251:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.381277:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381281:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.381291:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381294:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.381302:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.381317:0:327572:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8195b00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431515.381661:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431515.381695:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431515.381767:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.381778:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431515.381791:0:327572:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431515.381802:0:327572:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6a00 x1876350913503232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431515.381813:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.385781:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913503232:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.386079:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.391416:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913503232:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.391451:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391468:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391475:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391482:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391496:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391503:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.391508:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391515:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391522:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391530:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.391538:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.391545:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e774 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.391571:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6a00 x1876350913503232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.391589:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.392146:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.392167:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431515.392177:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b9a0 00010000:00010000:1.0:1789431515.392181:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.392190:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.394602:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394629:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.394652:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394659:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.394677:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394681:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.394699:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394702:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.394711:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394714:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.394724:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394727:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.394793:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394799:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.394816:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.394819:0:327572:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.394827:0:327572:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.395220:0:327572:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c4d13c80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.395232:0:327572:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.395284:0:327572:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.395468:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.395482:0:327572:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.395498:0:327572:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.395595:0:327572:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913503488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.395607:0:327572:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.395638:0:327572:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913503488:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.396104:0:327572:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.406101:0:327572:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327572:1876350913503488:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.406131:0:327572:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406147:0:327572:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406154:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406162:0:327572:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406171:0:327572:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406177:0:327572:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.406181:0:327572:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406188:0:327572:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406195:0:327572:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406201:0:327572:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.406207:0:327572:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.406218:0:327572:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e77b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327572 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.406240:0:327572:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdf480 x1876350913503488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.406264:0:327572:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.406306:0:327572:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.406321:0:327572:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.406343:0:327572:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c148b9a0 00010000:00010000:1.0:1789431515.406348:0:327572:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.406357:0:327572:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.414527:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431515.414554:0:327575:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431515.414564:0:327575:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431515.414594:0:327575:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431515.414619:0:327575:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.414640:0:327575:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.414650:0:327575:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431515.414662:0:327575:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431515.414667:0:327575:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e751 00000080:00010000:3.0:1789431515.414676:0:327575:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e751 from it ffffbc42c1c63cb8 00010000:00010000:3.0:1789431515.414681:0:327575:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.414693:0:327575:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431515.415565:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415582:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.415596:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415600:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.415607:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415610:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.415616:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415619:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.415626:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415630:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.415636:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415640:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.415648:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415651:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.415659:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415662:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.415673:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.415677:0:327574:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431515.416898:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.416938:0:327574:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:1.0:1789431515.416945:0:327574:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431515.416981:0:327574:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431515.417062:0:327574:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327574 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.417076:0:327574:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327574 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.417089:0:327574:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327574 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.417123:0:327574:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327574:1876350913503616:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:1.0:1789431515.417267:0:327574:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000080:00200000:3.0:1789431515.417909:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417922:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431515.417936:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417941:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.417951:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417954:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.417962:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417965:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.417974:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417977:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.417984:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.417988:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.417997:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.418001:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.418009:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.418012:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.418019:0:327575:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.418023:0:327575:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:1.0:1789431515.424179:0:327574:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327574:1876350913503616:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431515.424214:0:327574:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424245:0:327574:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424263:0:327574:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424276:0:327574:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424288:0:327574:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424303:0:327574:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424312:0:327574:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424321:0:327574:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.424356:0:327574:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdf480 x1876350913503616/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431515.424372:0:327574:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424384:0:327574:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1690 (0 0) 00000080:00200000:1.0:1789431515.424391:0:327574:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431515.424421:0:327574:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69a:0x0], updating i_size 0 00000080:00200000:1.0:1789431515.424425:0:327574:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x69a:0x0](ffff9811c4871988): rc = 0 00000080:00010000:1.0:1789431515.424430:0:327574:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x69a:0x0](ffff9811c4871988) for lock 0x1ffb2ee1bd27e782 00000080:00010000:1.0:1789431515.424440:0:327574:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x69a:0x0](ffff9811c4871988) 00000080:00010000:1.0:1789431515.424444:0:327574:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e782 from it ffffbc42c1c5be00 00010000:00010000:1.0:1789431515.424448:0:327574:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.424457:0:327574:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431515.424491:0:327574:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431515.424501:0:327574:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431515.424536:0:327574:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431515.424557:0:327574:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431515.424607:0:327574:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.424643:0:327574:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.424650:0:327574:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431515.424656:0:327574:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431515.424660:0:327574:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e782 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x69a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aca9a expref: -99 pid: 327574 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.424676:0:327574:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118800/0x1ffb2ee1bd27e751 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795aca00 expref: -99 pid: 327572 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431515.424695:0:327574:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431515.424701:0:327574:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdf480 x1876350913503872/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431515.424728:0:327574:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327574:1876350913503872:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431515.424861:0:327574:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.434554:0:327574:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327574:1876350913503872:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431515.434628:0:327574:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x69a:0x0](ffff9811c4871988) 00000080:00200000:1.0:1789431515.441718:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431515.441743:0:327577:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431515.441753:0:327577:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431515.441781:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431515.441798:0:327577:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:1.0:1789431515.441922:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.441933:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.441945:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.441978:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504000:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431515.442130:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.446490:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504000:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431515.446527:0:327577:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446541:0:327577:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446556:0:327577:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446568:0:327577:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446582:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446590:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446597:0:327577:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.446610:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdf800 x1876350913504000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431515.446627:0:327577:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.446644:0:327577:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431515.446652:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431515.446940:0:327577:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431515.446949:0:327577:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e789 00000080:00010000:0.0:1789431515.446956:0:327577:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431515.446960:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7bcb8 00010000:00010000:0.0:1789431515.446965:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.447264:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431515.449563:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.449589:0:327577:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431515.449594:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431515.449619:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431515.449679:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.449690:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.449700:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.449726:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504128:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431515.449813:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.455735:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504128:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.455773:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.455792:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.455799:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.455807:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.455818:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.455826:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.456289:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.456383:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.456392:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.456401:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.456410:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.456418:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27e790 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.456443:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc5180 x1876350913504128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.456460:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.456500:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.456517:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431515.456527:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b970 00010000:00010000:2.0:1789431515.456532:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.456547:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431515.456657:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.456752:0:327577:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.456759:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.456782:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.457529:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.457542:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.457555:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.457588:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504256:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.459572:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.478941:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504256:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.478972:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.478990:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479008:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479017:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479028:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479036:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.479042:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479048:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479056:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479064:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.479071:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479078:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e797 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.479103:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913504256/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.479120:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.479159:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.479174:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431515.479185:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b970 00010000:00010000:2.0:1789431515.479189:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.479198:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431515.479500:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.479512:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.479529:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.479533:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.479542:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.479545:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.479554:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.479566:0:327577:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.479571:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.479596:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.479650:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479659:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.479669:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.479696:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504384:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.480074:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.483210:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504384:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.483249:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483265:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483279:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483286:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483481:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483490:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.483495:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483502:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483511:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483519:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.483526:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.483533:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e79e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.483555:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913504384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.483573:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.483611:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431515.483626:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431515.483637:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b970 00010000:00010000:2.0:1789431515.483641:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431515.483650:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431515.483766:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.483777:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.484109:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.484114:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.484125:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.484141:0:327577:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b63b00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.484150:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.484173:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.484233:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.484241:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.484252:0:327577:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.484260:0:327577:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100a80 x1876350913504640/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.484274:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.484310:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504640:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.484579:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.488547:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504640:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.488578:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488593:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488601:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488609:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488618:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488625:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.488630:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488638:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488645:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488653:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.488661:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.488667:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7a5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.488689:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913504640/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.488706:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.488757:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431515.488773:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431515.488889:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b9a0 00010000:00010000:2.0:1789431515.488895:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431515.488904:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1789431515.489406:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489420:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.489434:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489438:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.489448:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489453:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.489462:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489465:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.489473:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489475:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.489483:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489487:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.489495:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489499:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.489508:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489510:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.489521:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.489532:0:327577:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62780, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.489539:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.489563:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.489628:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.489636:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.489649:0:327577:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.489659:0:327577:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100a80 x1876350913504896/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.489669:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.489695:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504896:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.490757:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.494014:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913504896:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.494048:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494066:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494074:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494082:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494094:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494101:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.494107:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494115:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494122:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494130:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.494137:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.494143:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ac lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.494172:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913504896/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.494189:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.494226:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431515.494241:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:2.0:1789431515.494251:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b9a0 00010000:00010000:2.0:1789431515.494254:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431515.494263:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431515.495653:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495738:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.495919:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495926:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.495940:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495943:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.495952:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495956:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.495965:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495968:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.495977:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495981:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.495990:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.495992:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.496002:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.496005:0:327577:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.496013:0:327577:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.496026:0:327577:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c2600, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.496033:0:327577:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.496066:0:327577:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.496149:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.496160:0:327577:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.496251:0:327577:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.496264:0:327577:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100a80 x1876350913505152/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.496275:0:327577:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.496384:0:327577:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913505152:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.496634:0:327577:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.502194:0:327577:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327577:1876350913505152:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.502226:0:327577:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502243:0:327577:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502255:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502262:0:327577:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502272:0:327577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502279:0:327577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.502284:0:327577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502376:0:327577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502386:0:327577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502394:0:327577:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.502401:0:327577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.502409:0:327577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e7b3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327577 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.502436:0:327577:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100a80 x1876350913505152/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.502453:0:327577:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.502602:0:327577:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431515.502619:0:327577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431515.502629:0:327577:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c7b9a0 00010000:00010000:0.0:1789431515.502634:0:327577:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431515.502642:0:327577:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431515.517494:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431515.517518:0:327580:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431515.517527:0:327580:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431515.517556:0:327580:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431515.517580:0:327580:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431515.517657:0:327580:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431515.517667:0:327580:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431515.517723:0:327580:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431515.517727:0:327580:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e789 00000080:00010000:2.0:1789431515.517738:0:327580:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e789 from it ffffbc42c1c93cb8 00010000:00010000:2.0:1789431515.517792:0:327580:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431515.517804:0:327580:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431515.520136:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520158:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.520172:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520178:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.520186:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520189:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.520197:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520200:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.520207:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520210:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.520221:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520225:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.520234:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520237:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.520244:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520246:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.520253:0:327580:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.520257:0:327580:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431515.522091:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522114:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431515.522127:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522132:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.522141:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522145:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.522154:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522157:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.522167:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522170:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.522181:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522185:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.522198:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522202:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.522213:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522217:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.522225:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.522229:0:327579:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431515.526003:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.526045:0:327579:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:3.0:1789431515.526051:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431515.526058:0:327579:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431515.526066:0:327579:0:(namei.c:2010:ll_mknod()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) mode 20644 dev 100002 00010000:00010000:3.0:1789431515.526135:0:327579:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431515.526172:0:327579:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431515.526281:0:327579:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431515.526289:0:327579:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431515.526293:0:327579:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27e789 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acabd expref: -99 pid: 327577 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1789431515.526317:0:327579:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431515.526344:0:327579:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc5f80 x1876350913505408/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 544/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431515.526384:0:327579:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327579:1876350913505408:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431515.527339:0:327579:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.533886:0:327579:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327579:1876350913505408:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431515.533950:0:327579:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x69c:0x0](ffff9811c4877448) 00000080:00200000:0.0:1789431515.533957:0:327579:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69c:0x0], updating i_size 0 00000080:00200000:2.0:1789431515.543425:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431515.543514:0:327582:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431515.543523:0:327582:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431515.543552:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431515.543571:0:327582:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431515.543647:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.543659:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.543672:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.543705:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505536:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431515.543827:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.549773:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505536:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431515.549811:0:327582:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549827:0:327582:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549878:0:327582:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549891:0:327582:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549906:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549916:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549924:0:327582:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.549939:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2100700 x1876350913505536/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431515.549956:0:327582:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.549973:0:327582:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431515.549981:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431515.550018:0:327582:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431515.550026:0:327582:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e7ba 00000080:00010000:2.0:1789431515.550032:0:327582:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431515.550035:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c63cb8 00010000:00010000:2.0:1789431515.550039:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.550048:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431515.551771:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.551793:0:327582:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.551800:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.551865:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.551942:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.551953:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.551964:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.551996:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505664:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.552114:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431515.558627:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505664:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431515.558658:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558676:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558696:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558707:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558718:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558726:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431515.558732:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558740:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558748:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558756:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431515.558764:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.558772:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62200/0x1ffb2ee1bd27e7c1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431515.558795:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913505664/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431515.558814:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.559012:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.559031:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431515.559043:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c63970 00010000:00010000:2.0:1789431515.559048:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431515.559057:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431515.559190:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.559215:0:327582:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.559220:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.559258:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.559316:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.559324:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.559348:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.559377:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505792:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.559512:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.566352:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505792:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431515.566385:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566400:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566413:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566421:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566431:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566440:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431515.566446:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566455:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566463:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566472:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431515.566480:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.566487:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e7c8 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431515.566510:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913505792/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431515.566527:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431515.566564:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431515.566580:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431515.566590:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c63970 00010000:00010000:3.0:1789431515.566594:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431515.566666:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431515.567162:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.567177:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.567197:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.567201:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.567211:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.567214:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.567219:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.567241:0:327582:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431515.567247:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431515.567274:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431515.567347:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.567359:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.567371:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431515.567398:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505920:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431515.567529:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.572612:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913505920:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431515.572646:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572664:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572671:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572679:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572689:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572696:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431515.572703:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572711:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572719:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.572727:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431515.573104:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.573114:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e7cf lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431515.573139:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7480 x1876350913505920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431515.573156:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.574666:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431515.574691:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431515.574707:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c63970 00010000:00010000:2.0:1789431515.574714:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431515.574723:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431515.575200:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.575211:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.575226:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.576221:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.576277:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.576310:0:327582:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c3c80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.576317:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.576361:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.576446:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.576456:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.576470:0:327582:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.576486:0:327582:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102d80 x1876350913506176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.576497:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.576528:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506176:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.577174:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.588867:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506176:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.588900:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588917:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588930:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588937:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588947:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588954:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.588959:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588967:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588974:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588981:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.588987:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.588994:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e7d6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.589016:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102d80 x1876350913506176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.589033:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.589071:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.589088:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:0.0:1789431515.589099:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c639a0 00010000:00010000:0.0:1789431515.589103:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.589112:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431515.593543:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593568:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.593587:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593591:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.593598:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593601:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.593608:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593611:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.593619:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593621:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.593629:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593634:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.593643:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593646:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.593655:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593658:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.593666:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.593684:0:327582:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8195b00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431515.593694:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431515.593731:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431515.594636:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.594651:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431515.594666:0:327582:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431515.594681:0:327582:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6680 x1876350913506432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431515.594692:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.594720:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506432:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.598736:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.608707:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506432:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.608753:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608771:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608780:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608788:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608804:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608812:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.608818:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608827:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608893:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608901:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.608909:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.608916:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e7dd lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.608940:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6680 x1876350913506432/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.608958:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.608994:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431515.609009:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:0.0:1789431515.609019:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c639a0 00010000:00010000:0.0:1789431515.609025:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431515.609034:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431515.609889:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609904:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.609919:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609923:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.609932:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609936:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.609944:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609948:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.609956:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609959:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.609967:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609972:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.609981:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609984:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.609992:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.609995:0:327582:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.610003:0:327582:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.610020:0:327582:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c2c00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.610029:0:327582:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.610058:0:327582:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.610131:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.610142:0:327582:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.610161:0:327582:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.610179:0:327582:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102a00 x1876350913506688/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.610192:0:327582:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.610218:0:327582:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506688:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.611246:0:327582:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.619520:0:327582:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327582:1876350913506688:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.619554:0:327582:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619571:0:327582:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619579:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619588:0:327582:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619598:0:327582:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619605:0:327582:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.619611:0:327582:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619619:0:327582:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619627:0:327582:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619636:0:327582:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.619644:0:327582:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.619651:0:327582:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e7e4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327582 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.619673:0:327582:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102a00 x1876350913506688/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.619690:0:327582:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.619729:0:327582:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431515.619744:0:327582:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431515.619756:0:327582:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c639a0 00010000:00010000:0.0:1789431515.619760:0:327582:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431515.619768:0:327582:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431515.636708:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431515.636746:0:327585:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431515.636757:0:327585:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431515.636792:0:327585:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431515.636819:0:327585:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431515.636869:0:327585:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431515.636879:0:327585:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431515.636890:0:327585:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431515.636894:0:327585:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e7ba 00000080:00010000:0.0:1789431515.636903:0:327585:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7ba from it ffffbc42c1c43cb8 00010000:00010000:0.0:1789431515.636908:0:327585:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431515.636918:0:327585:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431515.638556:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638575:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.638591:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638596:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.638608:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638611:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.638623:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638626:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.638637:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638641:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.638649:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638653:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.638660:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638664:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.638671:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638674:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.638682:0:327585:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.638686:0:327585:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431515.652091:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652115:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.652130:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652134:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.652142:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652145:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.652153:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652156:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.652165:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652169:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.652178:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652182:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.652190:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652193:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.652201:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652204:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.652212:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.652216:0:327584:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431515.656272:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.656325:0:327584:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431515.656340:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431515.656347:0:327584:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431515.656351:0:327584:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:2.0:1789431515.656413:0:327584:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431515.656445:0:327584:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431515.656551:0:327584:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431515.656561:0:327584:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431515.656565:0:327584:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7ba lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acb57 expref: -99 pid: 327582 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:2.0:1789431515.656596:0:327584:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431515.656618:0:327584:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101500 x1876350913506944/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431515.656652:0:327584:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327584:1876350913506944:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431515.656802:0:327584:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.663319:0:327584:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327584:1876350913506944:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431515.671908:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431515.671938:0:327587:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431515.671947:0:327587:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431515.671977:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431515.671997:0:327587:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431515.672074:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.672086:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.672098:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.672134:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507072:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431515.672265:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.690049:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507072:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431515.690087:0:327587:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690102:0:327587:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690117:0:327587:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690129:0:327587:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690143:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690152:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690160:0:327587:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.690173:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2103480 x1876350913507072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431515.690189:0:327587:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690203:0:327587:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431515.690211:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431515.690247:0:327587:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431515.690253:0:327587:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e7eb 00000080:00010000:0.0:1789431515.690258:0:327587:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431515.690263:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c43cb8 00010000:00010000:0.0:1789431515.690266:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.690276:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431515.696154:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.696183:0:327587:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.696203:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.698540:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.698769:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.698782:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.698793:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.698826:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507200:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.700653:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.709086:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507200:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.709122:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709140:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709148:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709155:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709167:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709174:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.709179:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709187:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709195:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709204:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.709211:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709219:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e7f2 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.709242:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913507200/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.709261:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.709299:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.709314:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431515.709324:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c43970 00010000:00010000:1.0:1789431515.709340:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431515.709349:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431515.709518:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.709536:0:327587:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.709659:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.709685:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.709739:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709748:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.709760:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.709786:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507328:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.710306:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.722204:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507328:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.722237:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722256:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722264:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722273:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722285:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722292:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.722297:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722305:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722314:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722322:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.722341:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722348:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e7f9 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.722371:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913507328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.722388:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.722424:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.722496:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431515.722506:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c43970 00010000:00010000:1.0:1789431515.722509:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431515.722517:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431515.722656:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.722666:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.722679:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.722705:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.722713:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.722716:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.722722:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.722732:0:327587:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.722736:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.722758:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.722805:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722811:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.722818:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.722982:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507456:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.723115:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.731100:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507456:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.731133:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731151:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731158:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731166:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731176:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731184:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.731190:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731197:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731205:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731213:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.731221:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731228:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e800 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.731256:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913507456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.731273:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.731310:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.731324:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431515.731346:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c43970 00010000:00010000:1.0:1789431515.731350:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.731359:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431515.731455:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.731466:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.731479:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.731483:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.731492:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.731511:0:327587:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65c80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.731519:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.731542:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.731598:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.731611:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.731624:0:327587:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.731631:0:327587:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913507712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.731640:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.731665:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507712:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.732066:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.740283:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507712:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.740316:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740342:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740350:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740358:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740369:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740376:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.740382:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740390:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740398:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740405:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.740413:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.740419:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e807 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.740442:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913507712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.740459:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.740497:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.740513:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431515.740522:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.740526:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431515.740535:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431515.742033:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742055:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.742072:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742076:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.742085:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742093:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.742105:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742108:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.742115:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742118:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.742127:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742131:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.742140:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742144:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.742152:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742156:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.742164:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.742179:0:327587:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64600, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.742186:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.742220:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.742300:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.742310:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.742324:0:327587:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.742907:0:327587:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913507968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.742922:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.742953:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507968:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.743252:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.753740:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913507968:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.753775:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.753792:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.753800:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.753809:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.753819:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.753826:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.756379:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.756403:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.756415:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.756426:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.756438:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.756447:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e80e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.756486:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913507968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.756503:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.756624:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.756641:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431515.756650:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.756654:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431515.757174:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.757674:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757688:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.757701:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757706:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.757716:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757725:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.757736:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757739:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.757748:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757751:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.757760:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757763:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.757773:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757776:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.757784:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757787:0:327587:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.757797:0:327587:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.757810:0:327587:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64000, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.757818:0:327587:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.757958:0:327587:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.758046:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.758055:0:327587:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.758070:0:327587:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.758084:0:327587:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913508224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.758094:0:327587:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.758120:0:327587:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913508224:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.758451:0:327587:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.762023:0:327587:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327587:1876350913508224:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.762058:0:327587:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762078:0:327587:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762087:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762096:0:327587:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762108:0:327587:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762117:0:327587:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.762124:0:327587:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762133:0:327587:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762142:0:327587:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762150:0:327587:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.762159:0:327587:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.762166:0:327587:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e815 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327587 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.762200:0:327587:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913508224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.762217:0:327587:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.762271:0:327587:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.762287:0:327587:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.762299:0:327587:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1c439a0 00010000:00010000:1.0:1789431515.762303:0:327587:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.762314:0:327587:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.775675:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431515.775706:0:327590:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431515.775715:0:327590:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431515.775757:0:327590:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431515.775782:0:327590:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.775800:0:327590:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431515.775810:0:327590:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431515.775822:0:327590:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431515.775826:0:327590:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e7eb 00000080:00010000:3.0:1789431515.775854:0:327590:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e7eb from it ffffbc42c1cabcb8 00010000:00010000:3.0:1789431515.775859:0:327590:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431515.775869:0:327590:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431515.777534:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777554:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431515.777572:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777577:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.777590:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777593:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.777604:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777607:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.777617:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777620:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431515.777629:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777634:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.777641:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777644:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.777652:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777654:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431515.777662:0:327590:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431515.777666:0:327590:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.779577:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779599:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.779614:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779618:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.779629:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779633:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.779642:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779645:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.779654:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779657:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.779667:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779670:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.779681:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779685:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.779693:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779697:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.779706:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.779711:0:327589:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431515.781538:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.781573:0:327589:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:0.0:1789431515.781580:0:327589:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:0.0:1789431515.781673:0:327589:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1789431515.781753:0:327589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327589 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.781766:0:327589:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327589 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.781778:0:327589:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327589 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.781812:0:327589:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327589:1876350913508352:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:0.0:1789431515.782425:0:327589:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.795258:0:327589:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327589:1876350913508352:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:0.0:1789431515.795310:0:327589:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795348:0:327589:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795359:0:327589:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795373:0:327589:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795386:0:327589:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795401:0:327589:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795409:0:327589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795417:0:327589:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.795432:0:327589:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd6680 x1876350913508352/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431515.795447:0:327589:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795459:0:327589:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1692 (0 0) 00000080:00200000:0.0:1789431515.795467:0:327589:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1789431515.795499:0:327589:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69c:0x0], updating i_size 0 00000080:00200000:0.0:1789431515.795514:0:327589:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x69c:0x0](ffff9811c4877448): rc = 0 00000080:00010000:0.0:1789431515.795520:0:327589:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x69c:0x0](ffff9811c4877448) for lock 0x1ffb2ee1bd27e81c 00000080:00010000:0.0:1789431515.795532:0:327589:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x69c:0x0](ffff9811c4877448) 00000080:00010000:0.0:1789431515.795538:0:327589:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e81c from it ffffbc42c1c9be00 00010000:00010000:0.0:1789431515.795541:0:327589:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.795551:0:327589:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431515.795589:0:327589:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:0.0:1789431515.795601:0:327589:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:0.0:1789431515.795637:0:327589:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.795660:0:327589:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:0.0:1789431515.795727:0:327589:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.795768:0:327589:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 1/0,0 mode: --/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.795776:0:327589:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431515.795783:0:327589:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1789431515.795786:0:327589:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e81c lrc: 0/0,0 mode: --/PR res: [0x200000402:0x69c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acc7d expref: -99 pid: 327589 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.795803:0:327589:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27e7eb lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acbe3 expref: -99 pid: 327587 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:0.0:1789431515.795822:0:327589:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431515.795884:0:327589:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6680 x1876350913508608/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431515.795912:0:327589:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327589:1876350913508608:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431515.796033:0:327589:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.805163:0:327589:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327589:1876350913508608:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431515.805234:0:327589:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x69c:0x0](ffff9811c4877448) 00000080:00200000:0.0:1789431515.811826:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431515.812005:0:327592:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431515.812016:0:327592:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431515.812048:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431515.812067:0:327592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431515.812144:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.812156:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.812167:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.812201:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508736:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431515.814761:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00010000:00010000:3.0:1789431515.814879:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431515.820081:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508736:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431515.820118:0:327592:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820138:0:327592:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820151:0:327592:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820164:0:327592:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820178:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820188:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820196:0:327592:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.820209:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7b80 x1876350913508736/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431515.820225:0:327592:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820241:0:327592:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431515.820249:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431515.820287:0:327592:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431515.820306:0:327592:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e823 00000080:00010000:0.0:1789431515.820313:0:327592:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431515.820316:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c53cb8 00010000:00010000:0.0:1789431515.820320:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.820337:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431515.824174:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.824203:0:327592:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.824208:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.824256:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.824612:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.824626:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.824637:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.824669:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508864:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.824890:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.830684:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508864:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.830720:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830737:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830747:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830755:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830765:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830772:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.830777:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830785:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830793:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830802:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.830809:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.830816:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e82a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.831064:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913508864/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.831084:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.831123:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.831139:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431515.831150:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.831154:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.831164:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431515.831420:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.831441:0:327592:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.831446:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.831467:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.831593:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.831602:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.831613:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.831642:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508992:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.831809:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.835087:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913508992:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.835119:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835136:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835143:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835151:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835161:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835168:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.835174:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835182:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835190:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835198:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.835207:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835213:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e831 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.835252:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913508992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.835273:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.835321:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431515.835347:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1789431515.835357:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.835361:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431515.835371:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431515.835467:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.835478:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.835493:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.835496:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.835523:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.835527:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.835539:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.835557:0:327592:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.835562:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.835583:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.835636:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835645:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.835655:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.835680:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509120:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.835780:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.840678:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509120:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.840709:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840727:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840736:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840743:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840754:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840762:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.840767:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840774:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840781:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840789:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.840795:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.840802:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e838 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.840822:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913509120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.840887:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.840925:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431515.841073:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1789431515.841087:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c53970 00010000:00010000:0.0:1789431515.841091:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431515.841100:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431515.841198:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.841209:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.841223:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.841227:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.841253:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.841268:0:327592:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0fb00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431515.841276:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431515.841450:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431515.841511:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.841520:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431515.841532:0:327592:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431515.841540:0:327592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7b80 x1876350913509376/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431515.841550:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.841576:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509376:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.842077:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.847315:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509376:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.847358:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847375:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847383:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847389:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847397:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847402:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.847406:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847411:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847417:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847421:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.847429:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.847436:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e83f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.847460:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913509376/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.847477:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.847515:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.847530:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:0.0:1789431515.847541:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c539a0 00010000:00010000:0.0:1789431515.847547:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431515.847561:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431515.848137:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848151:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431515.848177:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848182:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.848192:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848195:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.848203:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848205:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.848214:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848217:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431515.848224:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848228:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.848236:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848240:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.848249:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848251:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431515.848259:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.848272:0:327592:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0e480, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431515.848279:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431515.848307:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431515.848381:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.848391:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431515.848405:0:327592:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431515.848415:0:327592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7b80 x1876350913509632/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431515.848425:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.848452:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509632:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.848691:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.853527:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509632:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.853561:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853578:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853586:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853595:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853606:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853613:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.853618:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853631:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853639:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853647:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.853655:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.853662:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e846 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.853686:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913509632/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.853703:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.853742:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431515.853758:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:0.0:1789431515.853769:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c539a0 00010000:00010000:0.0:1789431515.853773:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431515.853782:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431515.854645:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854662:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431515.854676:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854680:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.854689:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854692:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.854700:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854703:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.854719:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854722:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431515.854731:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854735:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.854743:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854746:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.854754:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854757:0:327592:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.854765:0:327592:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.854779:0:327592:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c64780, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.854787:0:327592:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.854817:0:327592:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.855047:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.855058:0:327592:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.855073:0:327592:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.855085:0:327592:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdc700 x1876350913509888/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.855095:0:327592:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.855199:0:327592:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509888:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.855520:0:327592:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.863277:0:327592:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327592:1876350913509888:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431515.863319:0:327592:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863345:0:327592:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863352:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863360:0:327592:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863370:0:327592:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863376:0:327592:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.863381:0:327592:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863389:0:327592:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863397:0:327592:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863405:0:327592:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.863412:0:327592:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.863420:0:327592:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e84d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327592 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.863445:0:327592:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913509888/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.863467:0:327592:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.863506:0:327592:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.863519:0:327592:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431515.863526:0:327592:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1c539a0 00010000:00010000:1.0:1789431515.863529:0:327592:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431515.863536:0:327592:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431515.879559:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879583:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431515.879595:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879599:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.879611:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879614:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.879622:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879625:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.879640:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879643:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.879652:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879655:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.879663:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.879666:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.881112:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431515.881130:0:327595:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431515.881139:0:327595:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431515.881170:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431515.881196:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431515.881214:0:327595:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431515.881224:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431515.881235:0:327595:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431515.881239:0:327595:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e823 00000080:00010000:1.0:1789431515.881250:0:327595:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e823 from it ffffbc42c1cb3cb8 00010000:00010000:1.0:1789431515.881253:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431515.881264:0:327595:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431515.881303:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.881312:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431515.881338:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.881344:0:327594:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431515.883320:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.883363:0:327594:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431515.883368:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431515.883385:0:327594:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431515.883391:0:327594:0:(namei.c:2010:ll_mknod()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) mode 140777 dev 0 00010000:00010000:2.0:1789431515.883447:0:327594:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431515.883480:0:327594:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431515.883578:0:327594:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431515.883586:0:327594:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431515.883590:0:327594:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3a00/0x1ffb2ee1bd27e823 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acca0 expref: -99 pid: 327592 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:2.0:1789431515.883611:0:327594:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431515.883629:0:327594:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101f80 x1876350913510144/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 544/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431515.883659:0:327594:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327594:1876350913510144:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431515.883800:0:327594:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000080:00200000:1.0:1789431515.885043:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000100:00100000:0.0:1789431515.888302:0:327594:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327594:1876350913510144:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431515.888388:0:327594:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x69e:0x0](ffff9811c4871148) 00000080:00200000:0.0:1789431515.888396:0:327594:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69e:0x0], updating i_size 0 00000080:00200000:1.0:1789431515.889108:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.889119:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.889151:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.889228:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.889240:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.889250:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.889277:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510272:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.889404:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.894913:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510272:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.894954:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.894971:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.894984:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.894992:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895002:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895009:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.895015:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895023:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895031:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895039:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.895047:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895054:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e854 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.895072:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510272/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.895089:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.895128:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.895220:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.895238:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.895243:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.895267:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.895325:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895347:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.895358:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.895384:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510400:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.895494:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.898624:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510400:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.898663:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898680:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898689:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898696:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898707:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898718:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.898724:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898731:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898740:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898747:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.898755:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.898762:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e85b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.898781:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510400/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.898799:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.899101:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.899200:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.899225:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.899230:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.899264:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.899345:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.899353:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.899364:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.899390:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510528:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.899500:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.906583:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510528:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.906625:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906642:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906650:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906658:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906669:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906676:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.906682:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906690:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906698:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906708:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.906715:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.906723:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e862 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.906743:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510528/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.906760:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.906795:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.907028:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.907047:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.907052:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.907078:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.907139:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.907160:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.907170:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.907198:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510656:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.907301:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.911742:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510656:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.911776:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.911796:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.911803:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.911811:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.911825:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.912051:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.912057:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.912065:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.912073:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.912081:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.912090:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.912098:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e600/0x1ffb2ee1bd27e869 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.912122:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510656/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.913783:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.913862:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.914045:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.914068:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.914074:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.914100:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.914243:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.914254:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.914265:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.914296:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510784:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.914933:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.921443:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510784:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.921474:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921489:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921497:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921504:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921516:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921523:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.921529:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921537:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921544:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921551:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.921558:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921563:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e870 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.921583:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.921601:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.921637:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.921726:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.921744:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.921749:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.921768:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.921929:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921938:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.921948:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.921970:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510912:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.922213:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.926591:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913510912:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.926624:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926643:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926652:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926659:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926670:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926676:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.926681:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926688:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926697:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926704:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.926712:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.926718:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e877 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.926740:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913510912/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.926756:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.926804:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.926936:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.926958:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.926963:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.926986:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.927048:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.927057:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.927067:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.927095:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511040:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.927203:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.933569:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511040:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.933599:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933616:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933623:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933631:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933641:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933648:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.933654:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933661:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933670:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933677:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.933684:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.933691:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e87e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.933716:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913511040/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.933733:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.933768:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431515.934083:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.934105:0:327595:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431515.934110:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431515.934135:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431515.934202:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.934211:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.934221:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.934248:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511168:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431515.934524:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431515.939101:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511168:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431515.939138:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939225:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939234:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939243:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939253:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939261:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431515.939267:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939275:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939426:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939437:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431515.939445:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.939452:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e885 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431515.939475:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdc700 x1876350913511168/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431515.939492:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.939675:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431515.939773:0:327595:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.940357:0:327595:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d32b4600, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431515.940372:0:327595:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431515.940404:0:327595:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431515.940478:0:327595:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.940493:0:327595:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431515.940507:0:327595:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431515.940518:0:327595:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103800 x1876350913511424/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431515.940528:0:327595:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.940629:0:327595:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511424:192.168.203.128@tcp:101:tail.0 00000100:00100000:2.0:1789431515.941219:0:327595:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.945760:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913511552:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431515.945915:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431515.945923:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913511680:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431515.946144:0:327595:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327595:1876350913511424:192.168.203.128@tcp:101:tail.0 00010000:00010000:0.0:1789431515.946169:0:327595:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946186:0:327595:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946193:0:327595:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946202:0:327595:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946212:0:327595:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946219:0:327595:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.946224:0:327595:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946232:0:327595:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946240:0:327595:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946253:0:327595:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.946260:0:327595:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.946267:0:327595:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62a00/0x1ffb2ee1bd27e88c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 327595 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.946289:0:327595:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2103800 x1876350913511424/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.946306:0:327595:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.946353:0:327595:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431515.958198:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431515.958228:0:327597:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431515.958237:0:327597:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431515.958270:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431515.958364:0:327597:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431515.958445:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.958457:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.958470:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.958505:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913511808:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431515.959164:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00010000:00080000:2.0:1789431515.959595:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc7480 x1876350913511680/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431531 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431515.959614:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913511680:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431515.959626:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913511552:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431515.966524:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913511808:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431515.966639:0:327597:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.966656:0:327597:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967277:0:327597:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967377:0:327597:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967394:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967404:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967412:0:327597:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.967429:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7b80 x1876350913511808/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431515.967446:0:327597:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967462:0:327597:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431515.967469:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431515.967511:0:327597:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431515.967518:0:327597:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e893 00000080:00010000:0.0:1789431515.967524:0:327597:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431515.967528:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc3cb8 00010000:00010000:0.0:1789431515.967531:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.967618:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431515.974205:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.974235:0:327597:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.974241:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.974277:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431515.974450:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.974463:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.974475:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431515.974508:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913511936:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431515.974916:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.978520:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913511936:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.978552:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978577:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978585:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978593:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978603:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978610:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.978615:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978623:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978632:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978640:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.978648:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.978655:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e89a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.978683:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913511936/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.978708:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431515.978745:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.978759:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431515.978770:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc3970 00010000:00010000:0.0:1789431515.978774:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431515.978784:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431515.978923:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431515.978942:0:327597:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431515.978947:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431515.978970:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.979342:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.979354:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.979365:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.979392:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512064:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.979492:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431515.989046:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512064:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431515.989080:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989097:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989105:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989117:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989129:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989137:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431515.989142:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989151:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989172:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989182:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431515.989190:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431515.989199:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27e8a1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431515.989222:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913512064/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431515.989240:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431515.990070:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.990094:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431515.990106:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc3970 00010000:00010000:2.0:1789431515.990111:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431515.990120:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431515.990221:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.990232:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.990247:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.990251:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.990260:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.990264:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431515.990272:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431515.990285:0:327597:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431515.990290:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431515.990314:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431515.990378:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.990387:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431515.990398:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431515.990426:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512192:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431515.990523:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:0.0:1789431515.995572:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512192:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431515.995612:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995632:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995642:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995657:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995671:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995681:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431515.995689:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995699:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995712:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995722:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431515.995751:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431515.995763:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27e8a8 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431515.995792:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100000 x1876350913512192/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431515.995816:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431515.997921:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.997947:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431515.997957:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc3970 00010000:00010000:1.0:1789431515.997964:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431515.998228:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431515.998365:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.998377:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.998392:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.998395:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431515.998424:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431515.998820:0:327597:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431515.998866:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431515.998899:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431515.998993:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431515.999003:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431515.999018:0:327597:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431515.999034:0:327597:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913512448/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431515.999045:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431515.999083:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512448:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431515.999353:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.003145:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512448:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.003242:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003261:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003269:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003278:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003301:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003310:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.003316:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003324:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003341:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003350:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.003358:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.003365:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8af lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.003389:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913512448/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431531 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.003406:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.003692:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.003710:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431516.003720:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc39a0 00010000:00010000:1.0:1789431516.003724:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.003734:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431516.007744:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007768:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.007798:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007803:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.007814:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007818:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.007828:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007857:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.007871:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007875:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.007884:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007889:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.007899:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007902:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.007912:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007915:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.007923:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.007938:0:327597:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c8c65500, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.007947:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.008515:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.008611:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.008622:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.008637:0:327597:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.008652:0:327597:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913512704/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.008663:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.008706:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512704:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.009719:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.015096:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512704:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.015131:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015149:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015157:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015232:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015243:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015251:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.015256:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015264:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015272:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015280:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.015360:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.015369:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e8b6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.015394:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913512704/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.015418:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.015525:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.015542:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431516.015552:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc39a0 00010000:00010000:1.0:1789431516.015557:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.015566:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431516.016525:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016542:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.016556:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016561:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.016573:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016576:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.016586:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016589:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.016598:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016602:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.016613:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016617:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.016628:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016631:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.016643:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016646:0:327597:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.016656:0:327597:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.016737:0:327597:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c2180, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.016747:0:327597:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.016930:0:327597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.017090:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.017101:0:327597:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.017116:0:327597:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.017128:0:327597:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913512960/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.017139:0:327597:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.017236:0:327597:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512960:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.018405:0:327597:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.023759:0:327597:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327597:1876350913512960:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.023792:0:327597:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023809:0:327597:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023817:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023825:0:327597:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023918:0:327597:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023926:0:327597:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.023932:0:327597:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023940:0:327597:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023947:0:327597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.023955:0:327597:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.023963:0:327597:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.024042:0:327597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8bd lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327597 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.024069:0:327597:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913512960/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.024087:0:327597:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.024126:0:327597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.024141:0:327597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431516.024152:0:327597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1cc39a0 00010000:00010000:1.0:1789431516.024156:0:327597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.024236:0:327597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431516.039825:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431516.039871:0:327600:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431516.039880:0:327600:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431516.039909:0:327600:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431516.039935:0:327600:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.039954:0:327600:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431516.039964:0:327600:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431516.039987:0:327600:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431516.039992:0:327600:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e893 00000080:00010000:1.0:1789431516.040002:0:327600:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e893 from it ffffbc42c1c63cb8 00010000:00010000:1.0:1789431516.040006:0:327600:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.040018:0:327600:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431516.047640:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.047663:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431516.047891:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.047898:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431516.047910:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.047914:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.047925:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.047929:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.047940:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.047943:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.047953:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.047956:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.047966:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048075:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048080:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.048093:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048096:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.048106:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048109:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.048119:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048123:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.048134:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048139:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.048148:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048152:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.048162:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048165:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.048176:0:327600:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.048180:0:327600:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431516.050397:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.050652:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.050664:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.050708:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.050715:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.050733:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.050740:0:327599:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431516.051767:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.051799:0:327599:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431516.051804:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431516.051810:0:327599:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431516.051815:0:327599:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:2.0:1789431516.051909:0:327599:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431516.051945:0:327599:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431516.052044:0:327599:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431516.052053:0:327599:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431516.052057:0:327599:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27e893 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acdf7 expref: -99 pid: 327597 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:2.0:1789431516.052081:0:327599:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431516.052098:0:327599:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103b80 x1876350913513216/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431516.052129:0:327599:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327599:1876350913513216:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431516.052242:0:327599:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.056790:0:327599:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327599:1876350913513216:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431516.064339:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431516.065447:0:327602:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431516.065464:0:327602:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431516.065501:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431516.065521:0:327602:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431516.065592:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.065604:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.065616:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.065651:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513344:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431516.065772:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.080903:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513344:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431516.080941:0:327602:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.080955:0:327602:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.080971:0:327602:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.080984:0:327602:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.080999:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.081007:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.081016:0:327602:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.081029:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2103b80 x1876350913513344/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431516.081046:0:327602:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.081062:0:327602:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431516.081069:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431516.081119:0:327602:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431516.081127:0:327602:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e8c4 00000080:00010000:2.0:1789431516.081133:0:327602:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431516.081137:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb3cb8 00010000:00010000:2.0:1789431516.081141:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.081150:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431516.083449:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.083475:0:327602:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431516.083481:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431516.083513:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431516.090457:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.090489:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.090510:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.090555:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513472:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.090686:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.099985:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513472:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431516.100017:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100034:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100043:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100051:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100061:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100068:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431516.100073:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100080:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100088:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100096:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431516.100103:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100109:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e8cb lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.100131:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2103b80 x1876350913513472/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431516.100148:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.100183:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431516.100199:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431516.100211:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb3970 00010000:00010000:2.0:1789431516.100215:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431516.100223:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431516.100317:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.100344:0:327602:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431516.100348:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431516.100368:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431516.100414:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100422:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.100432:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.100457:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513600:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.100561:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.111208:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513600:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431516.111247:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111265:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111274:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111282:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111504:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111513:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431516.111518:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111527:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111535:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111544:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431516.111551:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.111559:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27e8d2 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.111584:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2103b80 x1876350913513600/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431516.111601:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.111641:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431516.111658:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431516.114446:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb3970 00010000:00010000:2.0:1789431516.114468:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431516.114486:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431516.115356:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.115373:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.115389:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.115393:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.115403:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.115407:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.115416:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.115431:0:327602:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431516.115437:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431516.115469:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.115550:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.115561:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.115572:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.115606:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513728:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.115723:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.119410:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513728:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.119444:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119461:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119469:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119477:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119488:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119495:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.119502:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119509:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119517:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119524:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.119532:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.119539:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118000/0x1ffb2ee1bd27e8d9 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.119563:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc5180 x1876350913513728/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.119581:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.119620:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431516.119635:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1789431516.119646:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb3970 00010000:00010000:0.0:1789431516.119650:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431516.119659:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431516.120191:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.120205:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.120222:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.120226:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.120235:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.120250:0:327602:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2a52f00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431516.120258:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431516.120284:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431516.120359:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.120368:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431516.120380:0:327602:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431516.120389:0:327602:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7b80 x1876350913513984/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431516.120399:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.120423:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513984:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.120710:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.128363:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913513984:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.128400:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128418:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128428:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128436:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128447:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128454:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.128458:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128467:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128475:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128482:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.128491:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.128498:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e8e0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.128527:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913513984/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.128545:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.129886:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431516.129915:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:0.0:1789431516.129929:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb39a0 00010000:00010000:0.0:1789431516.129935:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431516.129945:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431516.130502:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130515:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431516.130530:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130534:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.130544:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130548:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.130557:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130560:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.130569:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130586:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.130599:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130603:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.130616:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130619:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.130628:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130632:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.130641:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.130655:0:327602:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0fe00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431516.130663:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431516.130694:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431516.130769:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.130780:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431516.130794:0:327602:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431516.130809:0:327602:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7b80 x1876350913514240/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431516.130821:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.130890:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913514240:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.131139:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.135262:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913514240:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.135297:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135313:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135321:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135342:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135353:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135361:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.135366:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135373:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135381:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135390:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.135397:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.135408:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3e00/0x1ffb2ee1bd27e8e7 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.135433:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7b80 x1876350913514240/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.135451:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.135486:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.135501:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431516.135511:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb39a0 00010000:00010000:1.0:1789431516.135515:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.135525:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431516.136308:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136335:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.136351:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136356:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.136365:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136368:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.136375:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136378:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.136385:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136389:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.136397:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136401:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.136408:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136411:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.136418:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136421:0:327602:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.136430:0:327602:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.136442:0:327602:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c2f00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.136450:0:327602:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.136478:0:327602:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.136557:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.136568:0:327602:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.136580:0:327602:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.136590:0:327602:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913514496/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.136600:0:327602:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.136624:0:327602:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913514496:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.136876:0:327602:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.143264:0:327602:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327602:1876350913514496:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.143299:0:327602:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143323:0:327602:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143342:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143351:0:327602:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143362:0:327602:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143369:0:327602:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.143375:0:327602:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143383:0:327602:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143391:0:327602:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143398:0:327602:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.143406:0:327602:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.143413:0:327602:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e8ee lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327602 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.143435:0:327602:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913514496/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.143456:0:327602:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.143492:0:327602:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.143507:0:327602:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431516.143517:0:327602:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1cb39a0 00010000:00010000:1.0:1789431516.143522:0:327602:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.143531:0:327602:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431516.158806:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158863:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431516.158878:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158882:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.158890:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158893:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.158902:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158905:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.158914:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158917:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.158925:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158929:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.158938:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158941:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.158948:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158951:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.158957:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.158962:0:327604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431516.161963:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.162006:0:327604:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:3.0:1789431516.162013:0:327604:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000080:00200000:0.0:1789431516.163549:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431516.163577:0:327605:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431516.163587:0:327605:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431516.163620:0:327605:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431516.163647:0:327605:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431516.163666:0:327605:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431516.163676:0:327605:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431516.163686:0:327605:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431516.163691:0:327605:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e8c4 00000080:00010000:0.0:1789431516.163701:0:327605:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8c4 from it ffffbc42c1c53cb8 00010000:00010000:0.0:1789431516.163706:0:327605:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431516.163716:0:327605:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000002:00010000:3.0:1789431516.165381:0:327604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431516.165482:0:327604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.165496:0:327604:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.165508:0:327604:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327604 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.165546:0:327604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327604:1876350913514624:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431516.165692:0:327604:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000080:00200000:0.0:1789431516.166323:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166348:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431516.166365:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166370:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.166381:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166384:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.166393:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166396:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.166403:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166406:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.166415:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166419:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.166428:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166431:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.166440:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166443:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.166452:0:327605:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.166456:0:327605:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:1.0:1789431516.175524:0:327604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327604:1876350913514624:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431516.175560:0:327604:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175598:0:327604:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175609:0:327604:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175623:0:327604:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175643:0:327604:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175659:0:327604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175668:0:327604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175677:0:327604:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.175691:0:327604:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4000 x1876350913514624/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431516.175706:0:327604:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175718:0:327604:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1694 (0 0) 00000080:00200000:1.0:1789431516.175726:0:327604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431516.175753:0:327604:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x69e:0x0], updating i_size 0 00000080:00200000:1.0:1789431516.175757:0:327604:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x69e:0x0](ffff9811c4871148): rc = 0 00000080:00010000:1.0:1789431516.175762:0:327604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x69e:0x0](ffff9811c4871148) for lock 0x1ffb2ee1bd27e8f5 00000080:00010000:1.0:1789431516.175774:0:327604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x69e:0x0](ffff9811c4871148) 00000080:00010000:1.0:1789431516.175779:0:327604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8f5 from it ffffbc42c1c9be00 00010000:00010000:1.0:1789431516.175782:0:327604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.175798:0:327604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431516.175869:0:327604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431516.175879:0:327604:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431516.175919:0:327604:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431516.175941:0:327604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431516.175996:0:327604:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.176034:0:327604:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.176042:0:327604:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431516.176048:0:327604:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431516.176052:0:327604:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27e8f5 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x69e:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acf1d expref: -99 pid: 327604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.176066:0:327604:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27e8c4 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ace83 expref: -99 pid: 327602 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431516.176089:0:327604:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431516.176097:0:327604:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdfb80 x1876350913514880/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431516.176125:0:327604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327604:1876350913514880:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431516.176209:0:327604:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.179147:0:327604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327604:1876350913514880:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431516.179223:0:327604:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x69e:0x0](ffff9811c4871148) 00000080:00200000:3.0:1789431516.204155:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431516.204188:0:327607:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431516.204199:0:327607:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431516.204232:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431516.204253:0:327607:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431516.204340:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.204352:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.204367:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.204403:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515008:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431516.204528:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d9c0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.215913:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515008:192.168.203.128@tcp:101:sh.0 00010000:00010000:2.0:1789431516.215955:0:327607:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.215971:0:327607:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.215985:0:327607:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.215998:0:327607:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.216013:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.216212:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.216222:0:327607:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.216237:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc5880 x1876350913515008/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431516.216254:0:327607:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.216271:0:327607:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:2.0:1789431516.216278:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431516.216439:0:327607:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431516.216450:0:327607:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e8fc 00000080:00010000:0.0:1789431516.216457:0:327607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431516.216460:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccbcb8 00010000:00010000:0.0:1789431516.216465:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.216474:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431516.220630:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.220661:0:327607:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431516.220667:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431516.220704:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.228362:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.228389:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.228410:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.228453:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515136:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.228582:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.233456:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515136:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.233633:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233652:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233660:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233668:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233677:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233683:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.233688:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233695:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233703:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233712:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.233719:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.233724:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119000/0x1ffb2ee1bd27e903 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.233746:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2103b80 x1876350913515136/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.233763:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.233801:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431516.233818:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:0.0:1789431516.233829:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb970 00010000:00010000:0.0:1789431516.233855:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431516.233865:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431516.233968:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.233989:0:327607:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431516.233994:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431516.234018:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431516.234084:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.234092:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.234103:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.234131:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515264:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.234233:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.237718:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515264:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.237751:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237767:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237775:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237783:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237793:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237800:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.237806:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237814:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237824:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237863:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.237871:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.237878:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e90a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.237901:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6300 x1876350913515264/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.237919:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.237954:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431516.237969:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1789431516.237979:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb970 00010000:00010000:0.0:1789431516.237983:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431516.237992:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431516.238086:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.238099:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.238115:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.238119:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.238130:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.238132:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.238147:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.238160:0:327607:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431516.238164:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431516.238186:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431516.238611:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.238623:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.238635:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.238663:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515392:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.238768:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.241691:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515392:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.241724:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241742:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241749:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241755:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241765:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241772:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.241777:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241784:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241791:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241799:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.241806:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.241812:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27e911 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.241872:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd6300 x1876350913515392/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.241890:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.241928:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431516.241944:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:0.0:1789431516.241955:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb970 00010000:00010000:0.0:1789431516.241960:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431516.241968:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431516.242074:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.242085:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.242099:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.242102:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.242111:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.242127:0:327607:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c6bf6a80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431516.242133:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431516.242160:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431516.242219:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.242227:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431516.242237:0:327607:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431516.242245:0:327607:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6300 x1876350913515648/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431516.242254:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.242279:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515648:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.242531:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.247755:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515648:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.247788:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247805:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247813:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247821:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247851:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247859:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.247865:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247872:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247880:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247888:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.247895:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.247903:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2800/0x1ffb2ee1bd27e918 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.247934:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6300 x1876350913515648/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.247954:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.247992:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431516.248377:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:0.0:1789431516.248389:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb9a0 00010000:00010000:0.0:1789431516.248393:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431516.248403:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431516.248988:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249003:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431516.249013:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249017:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.249029:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249032:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.249040:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249043:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.249052:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249054:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.249079:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249084:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.249094:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249097:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.249104:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249108:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.249116:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.249129:0:327607:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c3a20300, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431516.249137:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431516.249165:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431516.249235:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.249245:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431516.249258:0:327607:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431516.249270:0:327607:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6300 x1876350913515904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431516.249281:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.249313:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515904:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.249566:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.252370:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913515904:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.252402:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252419:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252432:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252440:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252453:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252461:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.252467:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252475:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252483:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252492:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.252501:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.252509:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e91f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.252537:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6300 x1876350913515904/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.252556:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.252595:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431516.252611:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:0.0:1789431516.252622:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb9a0 00010000:00010000:0.0:1789431516.252627:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431516.252636:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431516.253453:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253471:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431516.253487:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253492:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.253503:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253507:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.253516:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253520:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.253530:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253533:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.253543:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253547:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.253556:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253559:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.253567:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253570:0:327607:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.253580:0:327607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.253595:0:327607:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0e0e180, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431516.253603:0:327607:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431516.253632:0:327607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431516.253704:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.253714:0:327607:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431516.253727:0:327607:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431516.254142:0:327607:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd6300 x1876350913516160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431516.254156:0:327607:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.254186:0:327607:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913516160:192.168.203.128@tcp:101:grep.0 00000100:00100000:0.0:1789431516.255445:0:327607:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.267474:0:327607:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327607:1876350913516160:192.168.203.128@tcp:101:grep.0 00010000:00010000:0.0:1789431516.267508:0:327607:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267525:0:327607:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267533:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267541:0:327607:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267557:0:327607:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267564:0:327607:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431516.267570:0:327607:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267578:0:327607:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267586:0:327607:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267593:0:327607:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431516.267601:0:327607:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.267608:0:327607:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e926 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327607 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.267630:0:327607:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd6300 x1876350913516160/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431516.267647:0:327607:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431516.267685:0:327607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431516.267700:0:327607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:0.0:1789431516.267710:0:327607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1ccb9a0 00010000:00010000:0.0:1789431516.267715:0:327607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431516.267724:0:327607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431516.285686:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431516.285712:0:327610:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431516.285722:0:327610:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431516.285827:0:327610:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431516.285886:0:327610:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431516.285906:0:327610:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431516.285916:0:327610:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431516.285925:0:327610:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431516.285930:0:327610:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e8fc 00000080:00010000:0.0:1789431516.285940:0:327610:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e8fc from it ffffbc42c1c9bcb8 00010000:00010000:0.0:1789431516.285944:0:327610:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431516.285954:0:327610:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431516.289520:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.289543:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431516.289559:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.289564:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.289574:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.289577:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.289586:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292278:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292300:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431516.292318:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292322:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.292349:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292353:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.292363:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292367:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.292376:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292379:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431516.292388:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292392:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.292400:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292403:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.292411:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292415:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431516.292424:0:327610:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431516.292428:0:327610:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431516.293399:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.293440:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.293452:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.293463:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.293466:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.293477:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.293480:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.293488:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.293491:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.293500:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.293504:0:327609:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431516.294001:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.294021:0:327609:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431516.294026:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431516.294031:0:327609:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431516.294036:0:327609:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:2.0:1789431516.294094:0:327609:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431516.294129:0:327609:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431516.294217:0:327609:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431516.294224:0:327609:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431516.294227:0:327609:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27e8fc lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acf40 expref: -99 pid: 327607 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:2.0:1789431516.294248:0:327609:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431516.294263:0:327609:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101880 x1876350913516416/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431516.294295:0:327609:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327609:1876350913516416:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431516.296040:0:327609:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.311123:0:327609:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327609:1876350913516416:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431516.311190:0:327609:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6a0:0x0](ffff9811c48b21c8) 00000080:00200000:1.0:1789431516.311198:0:327609:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a0:0x0], updating i_size 4 00000080:00200000:0.0:1789431516.318704:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431516.318730:0:327612:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431516.318752:0:327612:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431516.318783:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431516.318802:0:327612:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431516.318913:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.318924:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.318936:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.318968:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516544:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431516.319075:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.324030:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516544:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431516.324069:0:327612:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324085:0:327612:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324100:0:327612:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324113:0:327612:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324133:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324145:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324153:0:327612:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.324167:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd6300 x1876350913516544/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431516.324184:0:327612:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324200:0:327612:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:1.0:1789431516.324208:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1789431516.324246:0:327612:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1789431516.324253:0:327612:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e92d 00000080:00010000:1.0:1789431516.324259:0:327612:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431516.324263:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdbcb8 00010000:00010000:1.0:1789431516.324266:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.324277:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431516.326205:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.326231:0:327612:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:2.0:1789431516.326238:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:2.0:1789431516.326268:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431516.328180:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.328202:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.328223:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.328265:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516672:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.328433:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059240 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.339492:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516672:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.339525:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339542:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339550:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339559:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339569:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339576:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.339582:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339590:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339599:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339607:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.339613:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.339620:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27e934 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.339643:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2101c00 x1876350913516672/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.339667:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.339706:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.339721:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431516.339737:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb970 00010000:00010000:1.0:1789431516.339741:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.339752:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431516.339891:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.339912:0:327612:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431516.339917:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431516.339939:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431516.339993:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.340002:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.340012:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.340037:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516800:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.340214:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.345960:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516800:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.345993:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346011:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346020:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346029:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346047:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346054:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.346059:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346067:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346075:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346082:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.346089:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346100:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e93b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.346122:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913516800/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.346140:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.346177:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431516.346193:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431516.346203:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb970 00010000:00010000:1.0:1789431516.346207:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431516.346216:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431516.346458:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.346470:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.346488:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.346492:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.346501:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.346504:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.346514:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.346526:0:327612:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431516.346530:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431516.346550:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431516.346594:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346599:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.346606:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.346625:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516928:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.347280:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.350372:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913516928:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.350404:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350421:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350430:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350438:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350448:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350456:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.350461:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350469:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350477:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350486:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.350494:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.350500:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e942 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.350522:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913516928/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.350540:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.350583:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431516.350599:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431516.350609:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb970 00010000:00010000:1.0:1789431516.350613:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431516.350622:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431516.351358:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.351375:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.351389:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.351392:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.351402:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.351417:0:327612:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d33c2a80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.351425:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.351452:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.351515:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.351524:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.351538:0:327612:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.351547:0:327612:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913517184/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.351557:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.351581:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517184:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.351821:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.354908:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517184:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.354939:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.354957:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.354965:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.354972:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.354982:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.354990:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.354995:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.355002:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.355009:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.355016:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.355023:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.355030:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e949 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.355055:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913517184/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.355073:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.355110:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.355126:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431516.355136:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb9a0 00010000:00010000:1.0:1789431516.355140:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.355148:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431516.357705:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.357729:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.357746:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.357751:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.357762:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.357765:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.357773:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.357776:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.358883:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.358900:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.358915:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.358919:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.358928:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.358931:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.358947:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.358951:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.358960:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.358977:0:327612:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d32b4c00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.358987:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.359028:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.359124:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.359136:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.359150:0:327612:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.359166:0:327612:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913517440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.359175:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.359201:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517440:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.359529:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.364426:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517440:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.364466:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364486:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364494:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364502:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364519:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364527:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.364532:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364541:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364549:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364557:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.364566:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.364574:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e950 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.364599:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913517440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.364617:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.364655:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.364670:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431516.364752:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb9a0 00010000:00010000:1.0:1789431516.364758:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.364767:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431516.365459:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365474:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.365489:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365494:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.365506:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365509:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.365520:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365524:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.365533:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365536:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.365544:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365548:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.365557:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365561:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.365570:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365574:0:327612:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.365582:0:327612:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.365597:0:327612:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2a52000, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.365605:0:327612:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.365634:0:327612:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.366187:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.366199:0:327612:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.366214:0:327612:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.366228:0:327612:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913517696/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.366238:0:327612:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.366265:0:327612:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517696:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.366526:0:327612:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.370264:0:327612:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327612:1876350913517696:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.370319:0:327612:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370349:0:327612:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370357:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370365:0:327612:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370376:0:327612:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370384:0:327612:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.370389:0:327612:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370400:0:327612:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370408:0:327612:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370416:0:327612:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.370424:0:327612:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.370430:0:327612:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e957 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327612 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.370455:0:327612:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913517696/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.370473:0:327612:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.370511:0:327612:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.370525:0:327612:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:1.0:1789431516.370533:0:327612:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1cdb9a0 00010000:00010000:1.0:1789431516.370536:0:327612:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431516.370543:0:327612:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431516.379202:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431516.379230:0:327615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431516.379241:0:327615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431516.379274:0:327615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431516.379300:0:327615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.379319:0:327615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431516.379340:0:327615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431516.379352:0:327615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431516.379356:0:327615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e92d 00000080:00010000:1.0:1789431516.379368:0:327615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e92d from it ffffbc42c1ccbcb8 00010000:00010000:1.0:1789431516.379446:0:327615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.379458:0:327615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431516.381978:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382002:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.382018:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382023:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.382034:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382037:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.382048:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382051:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.382060:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382064:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.382073:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382077:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.382088:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382091:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.382099:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382102:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.382111:0:327615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.382115:0:327615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431516.382675:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382684:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431516.382694:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382698:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.382707:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382710:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.382719:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382723:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.382731:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382735:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.382744:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382747:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.382756:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382760:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.382770:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382773:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431516.382782:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.382785:0:327614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431516.383383:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.383407:0:327614:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:3.0:1789431516.383412:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431516.383418:0:327614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:3.0:1789431516.383423:0:327614:0:(namei.c:2084:ll_symlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), target=test 00010000:00010000:3.0:1789431516.383482:0:327614:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431516.383515:0:327614:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431516.383607:0:327614:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431516.383616:0:327614:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431516.383619:0:327614:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27e92d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795acfda expref: -99 pid: 327612 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:3.0:1789431516.383643:0:327614:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431516.383661:0:327614:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc6a00 x1876350913517952/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431516.383697:0:327614:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327614:1876350913517952:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431516.383824:0:327614:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.388750:0:327614:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327614:1876350913517952:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431516.399009:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431516.399038:0:327617:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431516.399048:0:327617:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431516.399261:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431516.399282:0:327617:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:3.0:1789431516.399364:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.399437:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.399451:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.399487:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518080:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431516.399589:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.403025:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518080:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431516.403065:0:327617:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403082:0:327617:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403096:0:327617:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403111:0:327617:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403126:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403135:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403145:0:327617:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431516.403157:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc6a00 x1876350913518080/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431516.403173:0:327617:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403188:0:327617:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431516.403196:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431516.403236:0:327617:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431516.403244:0:327617:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e95e 00000080:00010000:3.0:1789431516.403250:0:327617:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431516.403253:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d13cb8 00010000:00010000:3.0:1789431516.403257:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.403267:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431516.406109:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.406138:0:327617:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431516.406144:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431516.406178:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.406263:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.406277:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.406289:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.406335:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518208:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.407082:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.410412:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518208:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431516.410445:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410461:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410469:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410476:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410489:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410495:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431516.410501:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410508:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410515:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410522:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431516.410529:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410536:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e965 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431516.410559:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6a00 x1876350913518208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431516.410577:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431516.410612:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431516.410626:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431516.410636:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d13970 00010000:00010000:3.0:1789431516.410640:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431516.410648:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431516.410747:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.410764:0:327617:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431516.410770:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431516.410942:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.410991:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.410999:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.411007:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.411030:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518336:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.411116:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.414615:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518336:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431516.414649:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414665:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414683:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414691:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414701:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414707:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431516.414714:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414722:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414729:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414736:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431516.414744:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.414750:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27e96c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431516.414772:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6a00 x1876350913518336/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431516.414802:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431516.414875:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431516.414889:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431516.414899:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d13970 00010000:00010000:3.0:1789431516.414903:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431516.414911:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431516.415006:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.415016:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.415030:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.415034:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.415043:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.415060:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431516.415074:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.415087:0:327617:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431516.415092:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431516.415113:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.415165:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.415172:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.415181:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.415206:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518464:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.415299:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.419019:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518464:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431516.419051:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419066:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419079:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419087:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419096:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419103:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431516.419108:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419115:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419123:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419131:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431516.419138:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.419145:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27e973 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431516.419167:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc6a00 x1876350913518464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431516.419184:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.419388:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431516.419404:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431516.419416:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d13970 00010000:00010000:2.0:1789431516.419419:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431516.419434:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1789431516.419546:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.419558:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.419574:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.419577:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.419587:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.419604:0:327617:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62a80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431516.419612:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431516.419635:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431516.419770:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.420127:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431516.420146:0:327617:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431516.420156:0:327617:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102300 x1876350913518720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431516.420168:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.420199:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518720:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.420568:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.424170:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518720:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431516.424207:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424227:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424235:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424243:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424254:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424261:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431516.424266:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424274:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424282:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424375:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431516.424385:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.424392:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e97a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.424422:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102300 x1876350913518720/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431516.424440:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.424480:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431516.424496:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:2.0:1789431516.424507:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d139a0 00010000:00010000:2.0:1789431516.424511:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1789431516.424522:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1789431516.425558:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425577:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431516.425594:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425600:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.425611:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425615:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.425625:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425628:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.425637:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425640:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.425649:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425652:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.425661:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425663:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.425758:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.425763:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.425775:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.426071:0:327617:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0b62000, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431516.426080:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431516.426113:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431516.426194:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.426206:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431516.426220:0:327617:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431516.426235:0:327617:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102300 x1876350913518976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431516.426245:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.426270:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518976:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.426636:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:2.0:1789431516.431178:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913518976:192.168.203.128@tcp:101:grep.0 00010000:00010000:2.0:1789431516.431214:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431231:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431245:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431252:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431263:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431271:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431516.431275:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431283:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431303:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431312:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431516.431319:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.431335:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63c00/0x1ffb2ee1bd27e981 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431516.431361:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102300 x1876350913518976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431516.431378:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.431418:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431516.431434:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:2.0:1789431516.431444:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d139a0 00010000:00010000:2.0:1789431516.431449:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789431516.431458:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1789431516.432272:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432301:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431516.432319:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432333:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.432347:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432350:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.432360:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432363:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.432373:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432377:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.432385:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432389:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.432398:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432401:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.432408:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432411:0:327617:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.432419:0:327617:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.432432:0:327617:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0087b00, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431516.432441:0:327617:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431516.432469:0:327617:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431516.432544:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.432554:0:327617:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431516.432568:0:327617:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431516.432579:0:327617:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2102300 x1876350913519232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431516.432590:0:327617:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.432616:0:327617:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913519232:192.168.203.128@tcp:101:grep.0 00000100:00100000:2.0:1789431516.433202:0:327617:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000100:00100000:3.0:1789431516.437248:0:327617:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327617:1876350913519232:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431516.437280:0:327617:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437307:0:327617:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437314:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437331:0:327617:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437343:0:327617:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437350:0:327617:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431516.437356:0:327617:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437363:0:327617:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437372:0:327617:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437381:0:327617:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431516.437388:0:327617:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.437395:0:327617:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62600/0x1ffb2ee1bd27e988 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327617 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431516.437421:0:327617:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2102300 x1876350913519232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431516.437439:0:327617:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431516.437477:0:327617:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431516.437491:0:327617:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:3.0:1789431516.437501:0:327617:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1d139a0 00010000:00010000:3.0:1789431516.437505:0:327617:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431516.437514:0:327617:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431516.448128:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448154:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431516.448171:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448175:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.448185:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448188:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.448197:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448200:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.448209:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448212:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431516.448220:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448224:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.448234:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448238:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.448246:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448250:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431516.448258:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.448262:0:327619:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431516.449569:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431516.449600:0:327619:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=0 00000080:00200000:2.0:1789431516.449607:0:327619:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431516.449643:0:327619:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431516.450116:0:327619:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327619 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.450130:0:327619:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327619 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431516.450144:0:327619:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327619 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431516.450178:0:327619:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327619:1876350913519360:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431516.450534:0:327619:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d2059e40 going to sleep for 11 seconds 00000080:00200000:1.0:1789431516.453859:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431516.453887:0:327620:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431516.453897:0:327620:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431516.453928:0:327620:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431516.453953:0:327620:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.453972:0:327620:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431516.453982:0:327620:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431516.453991:0:327620:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431516.453995:0:327620:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e95e 00000080:00010000:1.0:1789431516.454004:0:327620:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e95e from it ffffbc42c1ccbcb8 00010000:00010000:1.0:1789431516.454007:0:327620:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431516.454023:0:327620:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431516.455503:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455523:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.455539:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455543:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.455555:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455558:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.455565:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455568:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.455575:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455578:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.455585:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455588:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.455595:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455597:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.455605:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455607:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.455618:0:327620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.455622:0:327620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:1.0:1789431516.460191:0:327619:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327619:1876350913519360:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431516.460227:0:327619:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460259:0:327619:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460274:0:327619:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460403:0:327619:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460418:0:327619:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460433:0:327619:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460443:0:327619:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460451:0:327619:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.460464:0:327619:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2101180 x1876350913519360/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431516.460478:0:327619:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460490:0:327619:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1696 (0 0) 00000080:00200000:1.0:1789431516.460499:0:327619:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431516.460527:0:327619:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a0:0x0], updating i_size 4 00000080:00200000:1.0:1789431516.460531:0:327619:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x6a0:0x0](ffff9811c48b21c8): rc = 0 00000080:00010000:1.0:1789431516.460536:0:327619:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a0:0x0](ffff9811c48b21c8) for lock 0x1ffb2ee1bd27e98f 00000080:00010000:1.0:1789431516.460548:0:327619:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a0:0x0](ffff9811c48b21c8) 00000080:00010000:1.0:1789431516.460551:0:327619:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e98f from it ffffbc42c1cdbe00 00010000:00010000:1.0:1789431516.460554:0:327619:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.460563:0:327619:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431516.460600:0:327619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:1.0:1789431516.460611:0:327619:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_f4670e3f0ec8ae778f90cebbfe2a1729, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:1.0:1789431516.460648:0:327619:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431516.461537:0:327619:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1789431516.461603:0:327619:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.461652:0:327619:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.461661:0:327619:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431516.461675:0:327619:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1789431516.461678:0:327619:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27e98f lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a0:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad100 expref: -99 pid: 327619 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.461695:0:327619:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27e95e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad066 expref: -99 pid: 327617 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000020:00100000:1.0:1789431516.461714:0:327619:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431516.461728:0:327619:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913519616/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 528/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431516.461760:0:327619:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327619:1876350913519616:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431516.462090:0:327619:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.465155:0:327619:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327619:1876350913519616:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431516.465228:0:327619:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x6a0:0x0](ffff9811c48b21c8) 00000080:00200000:0.0:1789431516.478579:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431516.478610:0:327622:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431516.478620:0:327622:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431516.478651:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431516.478726:0:327622:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:0.0:1789431516.478799:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.478811:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.478821:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431516.478929:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913519744:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431516.479086:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d094a780 going to sleep for 11 seconds 00000100:00100000:0.0:1789431516.481605:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913519744:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431516.481640:0:327622:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481656:0:327622:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481734:0:327622:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481749:0:327622:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481765:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481774:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481793:0:327622:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431516.481807:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7800 x1876350913519744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431516.481822:0:327622:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481873:0:327622:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:0.0:1789431516.481883:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431516.481919:0:327622:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431516.481927:0:327622:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431516.481933:0:327622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431516.481937:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d23cb8 00010000:00010000:0.0:1789431516.481940:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431516.481950:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431516.485145:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431516.485175:0:327622:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431516.485182:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431516.485217:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431516.485385:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.485399:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431516.485411:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431516.485447:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913519872:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431516.485608:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c450d6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.492735:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913519872:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.492768:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493178:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493189:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493197:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493207:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493219:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.493224:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493232:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493240:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493248:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.493255:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493262:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27e99d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.493280:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7100 x1876350913519872/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.493375:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.493416:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.493432:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431516.493443:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d23970 00010000:00010000:1.0:1789431516.493447:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431516.493462:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431516.493562:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.493580:0:327622:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431516.493586:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431516.493607:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431516.493660:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493740:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.493754:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.494002:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520000:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.494154:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.497719:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520000:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.497751:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497769:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497776:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497800:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497811:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497817:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.497821:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497828:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497900:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497908:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.497916:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.497922:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9a4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.497954:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913520000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.497971:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.498008:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431516.498023:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1789431516.498034:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d23970 00010000:00010000:1.0:1789431516.498039:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431516.498048:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431516.498140:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.498151:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.498163:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.498166:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.498176:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.498179:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.498187:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.498200:0:327622:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:1.0:1789431516.498205:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:1.0:1789431516.498226:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431516.498274:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.498282:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.498293:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.498319:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520128:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.498430:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.502913:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520128:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.502945:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.502962:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.502969:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.502976:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.502987:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.502994:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.503000:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.503007:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.503015:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.503023:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.503030:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.503036:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9ab lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.503064:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913520128/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.503079:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.503111:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431516.503122:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1789431516.503130:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d23970 00010000:00010000:1.0:1789431516.503133:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431516.503138:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431516.503222:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.503234:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.503248:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.503251:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.503260:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.503275:0:327622:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a92780, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.503282:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.503386:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.503446:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.503456:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.503465:0:327622:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.503471:0:327622:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde300 x1876350913520384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.503480:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.503505:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520384:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.503798:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.511283:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520384:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.511317:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511344:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511352:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511360:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511369:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511377:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.511383:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511390:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511398:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511406:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.511414:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.511421:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9b2 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.511445:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913520384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.511463:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.511498:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.511515:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431516.511523:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d239a0 00010000:00010000:1.0:1789431516.511526:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431516.511533:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431516.512079:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512093:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.512209:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512216:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.512230:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512233:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.512245:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512248:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.512256:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512258:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.512266:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512270:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.512278:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512281:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.512291:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512294:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.512303:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.512319:0:327622:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a92300, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.512336:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.512363:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.512429:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.512438:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.512451:0:327622:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.512462:0:327622:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde300 x1876350913520640/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.512473:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.512501:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520640:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.512760:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.519286:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520640:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.519319:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519346:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519355:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519363:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519373:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519381:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fa00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.519386:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519394:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519402:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519411:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.519419:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.519426:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fa00/0x1ffb2ee1bd27e9b9 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.519449:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913520640/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.519468:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431516.519505:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.519520:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:1.0:1789431516.519529:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d239a0 00010000:00010000:1.0:1789431516.519533:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431516.519541:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431516.522160:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522183:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431516.522198:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522203:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.522210:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522215:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.522222:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522225:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.522232:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522235:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431516.522242:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522246:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.522254:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522257:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.522266:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522268:0:327622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431516.522275:0:327622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431516.522292:0:327622:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a92900, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431516.522300:0:327622:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431516.522345:0:327622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431516.522441:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.522453:0:327622:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431516.522466:0:327622:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431516.522482:0:327622:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bde300 x1876350913520896/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431516.522492:0:327622:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431516.522519:0:327622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520896:192.168.203.128@tcp:101:grep.0 00000100:00100000:1.0:1789431516.523412:0:327622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431516.533674:0:327622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:327622:1876350913520896:192.168.203.128@tcp:101:grep.0 00010000:00010000:1.0:1789431516.533709:0:327622:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533726:0:327622:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533735:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533742:0:327622:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533753:0:327622:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533760:0:327622:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431516.533766:0:327622:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533774:0:327622:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533783:0:327622:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533790:0:327622:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431516.533799:0:327622:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431516.533805:0:327622:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27e9c0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 327622 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431516.533915:0:327622:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bde300 x1876350913520896/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431532 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431516.533933:0:327622:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431516.535647:0:327622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431516.535675:0:327622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:2.0:1789431516.535686:0:327622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d239a0 00010000:00010000:2.0:1789431516.535692:0:327622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1789431516.535708:0:327622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431516.838080:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431517.862995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431517.914815:0:327754:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:3.0:1789431518.886086:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431519.138570:0:327894:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:0.0:1789431519.911051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431520.933984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431521.065320:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913521024:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431521.065421:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913521280:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431521.065456:0:2389:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789431521.065462:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913521152:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431521.071795:0:2389:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9811d1e45800: committing for last_committed 4294974010 gen 1 00000100:00100000:1.0:1789431521.071888:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913521152:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431521.071902:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913521024:192.168.203.128@tcp:400:kworker.0 00010000:00080000:1.0:1789431521.071917:0:2389:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d0bdd880 x1876350913521280/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431537 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431521.071928:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913521280:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431521.959364:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431522.987443:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431524.007250:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431525.030258:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431525.198562:0:328470:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_09: symlink returns EPERM if the parent directory of the file named by name2 has its immutable flag set ========================================================== 20:18:44 (1789431524) 00010000:00010000:3.0:1789431526.058284:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789431526.184272:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913521408:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431526.184371:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913521664:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431526.184410:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789431526.184419:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913521536:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431526.196209:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913521408:192.168.203.128@tcp:400:kworker.0 00010000:00080000:0.0:1789431526.196235:0:2390:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d0bdf100 x1876350913521664/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431542 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431526.196248:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913521664:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431526.198321:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913521536:192.168.203.128@tcp:400:kworker.0 00000001:02000400:0.0:1789431526.554815:0:328609:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/09.t against ext4 filesystem 00010000:00010000:1.0:1789431527.078688:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431528.114708:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1789431528.818137:0:328760:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/09.t against lustre filesystem 00000080:00200000:0.0:1789431529.031119:0:328772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431529.031148:0:328772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431529.031167:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431529.031183:0:328772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431529.031195:0:328772:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431529.031203:0:328772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431529.031207:0:328772:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:0.0:1789431529.031213:0:328772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0da3b58 00010000:00010000:0.0:1789431529.031217:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431529.031225:0:328772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431529.031250:0:328772:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.031264:0:328772:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=13 00000080:00200000:0.0:1789431529.031269:0:328772:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:0.0:1789431529.031282:0:328772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1789431529.031324:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 328772 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.031330:0:328772:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 328772 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.031339:0:328772:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 328772 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.031363:0:328772:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328772:1876350913521792:192.168.203.128@tcp:101:lfs.0 00000100:00100000:0.0:1789431529.031555:0:328772:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28a80 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.036801:0:328772:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328772:1876350913521792:192.168.203.128@tcp:101:lfs.0 00010000:00010000:0.0:1789431529.036825:0:328772:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037126:0:328772:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037135:0:328772:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037146:0:328772:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037158:0:328772:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037172:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037180:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037187:0:328772:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.037199:0:328772:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7800 x1876350913521792/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431529.037212:0:328772:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037223:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431529.037306:0:328772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431529.037315:0:328772:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037321:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037327:0:328772:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.037348:0:328772:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1789431529.037354:0:328772:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1789431529.037380:0:328772:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:0.0:1789431529.037384:0:328772:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00000080:00010000:0.0:1789431529.037389:0:328772:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431529.037402:0:328772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431529.037406:0:328772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3b20 00010000:00010000:0.0:1789431529.037409:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431529.037418:0:328772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431529.037585:0:6161:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1789431529.037594:0:6161:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037616:0:6161:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431529.037622:0:6161:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037631:0:6161:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037639:0:6161:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037655:0:6161:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1789431529.037666:0:6161:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1789431529.037704:0:6161:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037711:0:6161:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:2.0:1789431529.037773:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913521920:192.168.203.128@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1789431529.037918:0:6161:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037926:0:6161:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9c7 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad1bd expref: -99 pid: 328772 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431529.037945:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431529.037947:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431529.037948:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1789431529.038408:0:328772:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431529.038417:0:328772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431529.038448:0:328772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.038460:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1789431529.038470:0:328772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1789431529.038477:0:328772:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.038483:0:328772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431529.038486:0:328772:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431529.038490:0:328772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:1.0:1789431529.038494:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1789431529.038502:0:328772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200008:1.0:1789431529.039051:0:328772:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431529.039056:0:328772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431529.039066:0:328772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.039076:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431529.039085:0:328772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431529.039091:0:328772:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.039096:0:328772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431529.039099:0:328772:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431529.039103:0:328772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0da3cb8 00010000:00010000:1.0:1789431529.039107:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431529.039114:0:328772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431529.039229:0:328772:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431529.039564:0:328772:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.039576:0:328772:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431529.039585:0:328772:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431529.039591:0:328772:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.039596:0:328772:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431529.039599:0:328772:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e4e9 00000080:00010000:1.0:1789431529.039603:0:328772:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e4e9 from it ffffbc42c0da3bb0 00010000:00010000:1.0:1789431529.039605:0:328772:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431529.039612:0:328772:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1789431529.039621:0:328772:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:1.0:1789431529.039628:0:328772:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:1.0:1789431529.039646:0:328772:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffc5f5461d0 00000080:00200000:1.0:1789431529.039659:0:328772:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:1.0:1789431529.039679:0:328772:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1789431529.039861:0:328772:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1789431529.039868:0:328772:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1789431529.039871:0:328772:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27e4e9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ac27b expref: -99 pid: 326955 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000020:00100000:1.0:1789431529.039895:0:328772:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431529.039906:0:328772:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913522176/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431529.039928:0:328772:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328772:1876350913522176:192.168.203.128@tcp:36:lfs.0 00000100:00100000:1.0:1789431529.039988:0:328772:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d3974300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431529.048176:0:328772:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328772:1876350913522176:192.168.203.128@tcp:36:lfs.0 00000080:00200008:1.0:1789431529.048291:0:328772:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000100:00100000:3.0:1789431529.049447:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913521920:192.168.203.128@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1789431529.125913:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1789431529.178395:0:328338:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431529.178407:0:328338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431529.179071:0:328338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431529.179134:0:328338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 328338 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.179144:0:328338:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 328338 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.179154:0:328338:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 328338 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.179179:0:328338:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328338:1876350913522304:192.168.203.128@tcp:101:bash.0 00000100:00100000:0.0:1789431529.179268:0:328338:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.189723:0:328338:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328338:1876350913522304:192.168.203.128@tcp:101:bash.0 00010000:00010000:0.0:1789431529.189752:0:328338:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189764:0:328338:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189776:0:328338:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189786:0:328338:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189798:0:328338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189806:0:328338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189815:0:328338:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.189827:0:328338:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd7800 x1876350913522304/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431529.189890:0:328338:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189903:0:328338:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1789431529.189910:0:328338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431529.189938:0:328338:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431529.189943:0:328338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431529.189948:0:328338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431529.189952:0:328338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1c53cb8 00010000:00010000:0.0:1789431529.189954:0:328338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.189963:0:328338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431529.190019:0:328338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431529.190035:0:328338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431529.190044:0:328338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431529.190054:0:328338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431529.190060:0:328338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431529.190064:0:328338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431529.190067:0:328338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431529.190069:0:328338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1c53b58 00010000:00010000:0.0:1789431529.190072:0:328338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431529.190079:0:328338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431529.190086:0:328338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.190093:0:328338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431529.190101:0:328338:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431529.190105:0:328338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431529.190116:0:328338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431529.190125:0:328338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1789431529.190133:0:328338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:0.0:1789431529.190140:0:328338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431529.190144:0:328338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431529.190148:0:328338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431529.190151:0:328338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c53cb8 00010000:00010000:0.0:1789431529.190155:0:328338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:0.0:1789431529.190164:0:328338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:0.0:1789431529.190188:0:328338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431529.190195:0:328338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431529.190201:0:328338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431529.190210:0:328338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431529.190216:0:328338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431529.190219:0:328338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431529.190222:0:328338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431529.190225:0:328338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1c53c18 00010000:00010000:0.0:1789431529.190228:0:328338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431529.190244:0:328338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431529.190253:0:328338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.190258:0:328338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:0.0:1789431529.190262:0:328338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:2.0:1789431529.195508:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195524:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431529.195538:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195542:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.195551:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195555:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.195563:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195567:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.195577:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195580:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.195589:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195592:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.195601:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195604:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.195613:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195616:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.195624:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.195634:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0ad4780, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431529.195641:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libperl.so.5.26, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431529.195666:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libperl.so.5.26,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431529.195733:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.195743:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431529.195755:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431529.195768:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101880 x1876350913522560/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431529.195780:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431529.195800:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913522560:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431529.196012:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec840 going to sleep for 11 seconds 00000100:00100000:2.0:1789431529.205325:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913522560:192.168.203.128@tcp:101:prove.0 00010000:00010000:2.0:1789431529.205350:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205363:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205370:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205377:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205386:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205396:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431529.205401:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205408:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205414:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205433:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431529.205440:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.205447:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27e9d5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431529.205464:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101880 x1876350913522560/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431529.205478:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431529.205504:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1789431529.205515:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00010000:2.0:1789431529.205524:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:2.0:1789431529.205528:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1789431529.205540:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1789431529.205961:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.205971:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431529.205982:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.205986:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.205993:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.205996:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206003:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206005:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206012:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206015:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206023:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206026:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206034:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206037:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206045:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206048:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206057:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206060:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431529.206193:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206198:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431529.206207:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206209:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206217:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206220:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206226:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206229:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206253:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206257:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.206266:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206268:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206276:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206279:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206287:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206289:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.206297:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.206307:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a93c80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431529.206313:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libresolv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431529.206331:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libresolv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431529.206378:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.206386:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431529.206396:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431529.206403:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101880 x1876350913522816/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431529.206413:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431529.206433:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913522816:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431529.206593:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec840 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.214758:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913522816:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.214781:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214795:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214802:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214810:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214818:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214825:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.214874:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214883:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214890:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214897:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.214903:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.214910:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27e9dc lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.214929:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101880 x1876350913522816/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.214949:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.214977:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1789431529.214990:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00010000:0.0:1789431529.214999:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:0.0:1789431529.215003:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:0.0:1789431529.215011:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:0.0:1789431529.215390:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.215401:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.215412:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.215417:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.215961:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.215969:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.215979:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.215983:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.215993:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.215997:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.216009:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.216014:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.216024:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.216027:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.216036:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.216039:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.216049:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.216060:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libdl.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c81f0c00, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431529.216067:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libdl.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431529.216088:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libdl.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431529.216140:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.216150:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431529.216160:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431529.216170:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913523072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431529.216179:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431529.216199:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523072:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431529.216382:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d3974300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431529.229670:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523072:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431529.229695:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229708:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229714:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229721:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229730:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229737:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431529.229741:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229748:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229755:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229760:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431529.229766:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.229772:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9e3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431529.229790:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913523072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431529.229804:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431529.229878:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1789431529.229892:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00010000:1.0:1789431529.229900:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:1.0:1789431529.229904:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1789431529.229911:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1789431529.230263:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230273:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431529.230285:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230291:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.230299:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230301:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.230308:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230311:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.230320:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230322:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.230331:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230334:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.230342:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230344:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.230353:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230355:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.230363:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.230373:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libm.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c81f1680, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431529.230379:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libm.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431529.230397:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libm.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431529.230458:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.230466:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431529.230475:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431529.230483:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913523328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431529.230492:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431529.230512:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523328:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431529.231483:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d3974300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431529.236905:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523328:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431529.236929:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236943:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236950:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236957:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236967:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236978:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431529.236984:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236990:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.236997:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.237004:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431529.237010:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.237016:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27e9ea lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431529.237034:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913523328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431529.237047:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431529.237075:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1789431529.237087:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00010000:1.0:1789431529.237096:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:1.0:1789431529.237100:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1789431529.237112:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1789431529.237470:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237480:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431529.237507:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237512:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.237522:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237525:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.237534:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237537:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.237548:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237551:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.237563:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237567:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.237576:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237579:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.237586:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237589:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.237597:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.237607:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0085080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431529.237613:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431529.237632:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431529.237677:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.237685:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431529.237696:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431529.237704:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bded80 x1876350913523584/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431529.237714:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431529.237734:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523584:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431529.237932:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d3974300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431529.244801:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523584:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431529.244825:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245116:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245125:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245132:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245141:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245148:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431529.245152:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245165:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245173:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245181:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431529.245271:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.245279:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27e9f1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431529.245300:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bded80 x1876350913523584/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431529.245314:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.245577:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1789431529.245592:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00010000:0.0:1789431529.245601:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:0.0:1789431529.245605:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1789431529.245619:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1789431529.246179:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246275:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.246291:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246295:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.246303:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246306:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.246313:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246316:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.246322:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246325:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.246332:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246335:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.246343:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246345:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.246352:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246354:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.246360:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.246370:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0082a80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431529.246376:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libutil.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431529.246394:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libutil.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431529.246441:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.246450:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431529.246461:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431529.246470:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913523840/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431529.246480:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.246571:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523840:192.168.203.128@tcp:101:prove.0 00000100:00100000:0.0:1789431529.246972:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.255881:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913523840:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.255916:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255930:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255938:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255945:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255953:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255959:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.255963:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255975:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255982:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.255989:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.255995:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.256001:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9f8 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.256018:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913523840/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.256032:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.256059:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1789431529.256073:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00010000:0.0:1789431529.256082:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:0.0:1789431529.256085:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:0.0:1789431529.256093:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1789431529.258279:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258294:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.258318:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258322:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.258332:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258334:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.258342:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258344:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.258353:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258356:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.258364:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258368:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.258376:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258378:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.258385:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258387:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.258394:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.258397:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431529.519152:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.519167:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.519180:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.519194:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.519205:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.519209:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.519217:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.519221:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.519229:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.519232:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.520073:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.520084:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.520099:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.520103:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.520112:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.520115:0:328775:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.520122:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.520133:0:328775:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0083c80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431529.520753:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=librt.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431529.520784:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: librt.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431529.521404:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.521415:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431529.521565:0:328775:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431529.521577:0:328775:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913524096/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431529.521586:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.521606:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524096:192.168.203.128@tcp:101:prove.0 00000100:00100000:0.0:1789431529.521897:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.531336:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524096:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.531360:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531374:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531382:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531389:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531397:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531404:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.531408:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531415:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.531422:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.532647:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.532658:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.532674:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27e9ff lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.532702:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913524096/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.532717:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.532748:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1789431529.532761:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00010000:0.0:1789431529.532769:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d879a0 00010000:00010000:0.0:1789431529.532773:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1789431529.532782:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1789431529.568707:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.568732:0:328775:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431529.568737:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.proverc, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431529.568761:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .proverc,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431529.568816:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.568826:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.568875:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431529.568901:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524224:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431529.569220:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec000 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.579567:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524224:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.579591:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579605:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579612:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579618:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579626:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579632:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.579636:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579644:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579655:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579662:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.579668:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.579674:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea06 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.579692:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2101f80 x1876350913524224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.579706:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.579733:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1789431529.579746:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00010000:0.0:1789431529.579754:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d87930 00010000:00010000:0.0:1789431529.579757:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1789431529.579765:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1789431529.582061:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.582090:0:328775:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:1.0:1789431529.582095:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=.prove, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:1.0:1789431529.582118:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: .prove,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431529.582170:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.582180:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.582188:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431529.582218:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524352:192.168.203.128@tcp:101:prove.0 00000100:00100000:1.0:1789431529.582321:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d3974300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431529.587278:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524352:192.168.203.128@tcp:101:prove.0 00010000:00010000:1.0:1789431529.587302:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587315:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587321:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587328:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587336:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587342:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431529.587346:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587352:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587358:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587364:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431529.587369:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431529.587380:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27ea0d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431529.587398:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d0bdfb80 x1876350913524352/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431529.587411:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431529.587437:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1789431529.587455:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00010000:1.0:1789431529.587465:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d87930 00010000:00010000:1.0:1789431529.587468:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1789431529.587476:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1789431529.690763:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.690778:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.690794:0:328775:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1 00000080:00200000:2.0:1789431529.690799:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=testrules.yml, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:2.0:1789431529.690822:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: testrules.yml,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431529.690981:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.690992:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431529.691001:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431529.691023:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524480:192.168.203.128@tcp:101:prove.0 00000100:00100000:2.0:1789431529.691111:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cec900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.708184:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524480:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.708220:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708243:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708250:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708257:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708266:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708271:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.708275:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708281:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708289:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708296:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.708301:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708311:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ea14 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.708329:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff9811d2100e00 x1876350913524480/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.708342:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.708367:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:0.0:1789431529.708379:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00010000:0.0:1789431529.708387:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d87930 00010000:00010000:0.0:1789431529.708391:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:0.0:1789431529.708398:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:0.0:1789431529.708461:0:328775:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.708479:0:328775:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:0.0:1789431529.708483:0:328775:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=t, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:0.0:1789431529.708499:0:328775:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: t,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:0.0:1789431529.708537:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708544:0:328775:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.708553:0:328775:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.708572:0:328775:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524608:192.168.203.128@tcp:101:prove.0 00000100:00100000:0.0:1789431529.708657:0:328775:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.712421:0:328775:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job prove:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328775:1876350913524608:192.168.203.128@tcp:101:prove.0 00010000:00010000:0.0:1789431529.712441:0:328775:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712454:0:328775:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712461:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712468:0:328775:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712476:0:328775:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712482:0:328775:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.712486:0:328775:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712494:0:328775:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712501:0:328775:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712508:0:328775:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.712513:0:328775:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.712520:0:328775:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea1b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328775 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.712535:0:328775:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913524608/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'prove.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.712549:0:328775:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.712574:0:328775:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1789431529.712587:0:328775:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00010000:0.0:1789431529.712596:0:328775:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d878d0 00010000:00010000:0.0:1789431529.712599:0:328775:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:0.0:1789431529.712607:0:328775:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:0.0:1789431529.913827:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913869:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.913886:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913891:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.913900:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913904:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.913913:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913916:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.913923:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913926:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.913936:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913939:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.913946:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913949:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.913956:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913959:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.913967:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.913970:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:0.0:1789431529.914144:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914149:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.914158:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914161:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.914170:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914173:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.914181:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914183:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.914190:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914193:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.914202:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914204:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.914212:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914215:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.914223:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914227:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.914718:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.914725:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431529.921389:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921405:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.921417:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921420:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921428:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921431:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921440:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921443:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921452:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921454:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921462:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921465:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921472:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921474:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921482:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921485:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921491:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921495:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:3.0:1789431529.921654:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921659:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.921668:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921671:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921680:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921683:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921692:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921695:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921703:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921706:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921712:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921715:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921723:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921725:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921733:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921736:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921744:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921748:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431529.921907:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921912:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.921921:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921924:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921931:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921934:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921941:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921943:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921952:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921956:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.921964:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921967:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921976:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921978:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921986:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921989:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.921996:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.921999:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431529.922118:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922124:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.922133:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922136:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922144:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922147:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922155:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922158:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922166:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922169:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922179:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922182:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922190:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922193:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922202:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922205:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922212:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922216:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:3.0:1789431529.922346:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922351:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.922362:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922365:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922374:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922377:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922385:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922388:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922396:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922399:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922406:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922409:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922417:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922420:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922427:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922430:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922436:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922440:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:3.0:1789431529.922567:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922572:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431529.922580:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922583:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922590:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922593:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922602:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922605:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922613:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922615:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431529.922625:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922628:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922637:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922639:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922647:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922650:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431529.922658:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431529.922663:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431529.931025:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931042:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.931055:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931059:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.931068:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931071:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.931082:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931085:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.931094:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931097:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.931104:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931107:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.931116:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931120:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.931126:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931129:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.931136:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.931146:0:328776:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libtinfo.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811ca227b00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431529.931153:0:328776:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libtinfo.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431529.931176:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libtinfo.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431529.931251:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.931261:0:328776:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431529.931272:0:328776:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431529.931284:0:328776:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913524864/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431529.931294:0:328776:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431529.931315:0:328776:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328776:1876350913524864:192.168.203.128@tcp:101:sh.0 00000100:00100000:0.0:1789431529.931494:0:328776:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431529.941702:0:328776:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328776:1876350913524864:192.168.203.128@tcp:101:sh.0 00010000:00010000:0.0:1789431529.941725:0:328776:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941739:0:328776:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941746:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941752:0:328776:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941761:0:328776:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941767:0:328776:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431529.941771:0:328776:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941777:0:328776:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941783:0:328776:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941790:0:328776:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431529.941796:0:328776:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431529.941802:0:328776:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea22 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328776 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431529.941817:0:328776:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913524864/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431545 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'sh.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431529.941865:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431529.941893:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1789431529.941905:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00010000:0.0:1789431529.941914:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0e2b9a0 00010000:00010000:0.0:1789431529.941918:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:0.0:1789431529.941925:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1789431529.942265:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942275:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.942290:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942293:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942300:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942303:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942311:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942314:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942322:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942325:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942333:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942336:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942345:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942348:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942356:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942359:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942367:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942370:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431529.942496:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942502:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431529.942511:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942513:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942521:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942524:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942532:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942534:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942541:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942543:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431529.942552:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942555:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942563:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942565:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942571:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942574:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431529.942580:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431529.942584:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431529.954539:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431529.954568:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.954586:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431529.954601:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431529.955726:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.955739:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431529.955743:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:1.0:1789431529.955752:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c0e2bb58 00010000:00010000:1.0:1789431529.955757:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431529.955769:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431529.955784:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.955793:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431529.955805:0:328776:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431529.955810:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431529.955979:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.955993:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1789431529.956004:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1789431529.956010:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.956015:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431529.956018:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431529.956022:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0e2bcb8 00010000:00010000:1.0:1789431529.956024:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:1.0:1789431529.956032:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1789431529.956052:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431529.956057:0:328776:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431529.956061:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431529.956069:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.956075:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1789431529.956083:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:1.0:1789431529.956089:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.956093:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431529.956095:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431529.956099:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0e2bcb8 00010000:00010000:1.0:1789431529.956102:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:1.0:1789431529.956110:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200008:1.0:1789431529.956137:0:328776:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431529.956141:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431529.956148:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.956154:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431529.956162:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431529.956169:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.956172:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431529.956175:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:1.0:1789431529.956178:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c0e2bcb8 00010000:00010000:1.0:1789431529.956181:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431529.956188:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431529.956203:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431529.956210:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.956215:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431529.956223:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431529.956228:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.956241:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431529.956244:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:1.0:1789431529.956247:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c0e2bb58 00010000:00010000:1.0:1789431529.956251:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431529.956257:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431529.956265:0:328776:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.956268:0:328776:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431529.956273:0:328776:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431529.956276:0:328776:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431529.956284:0:328776:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431529.956290:0:328776:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1789431529.956297:0:328776:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:1.0:1789431529.956303:0:328776:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431529.956305:0:328776:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431529.956307:0:328776:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431529.956310:0:328776:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0e2bcb8 00010000:00010000:1.0:1789431529.956312:0:328776:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:1.0:1789431529.956319:0:328776:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1789431529.966941:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431529.966962:0:328777:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431529.966970:0:328777:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431529.966997:0:328777:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431529.967016:0:328777:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1789431529.967031:0:328777:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:2.0:1789431529.967039:0:328777:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431529.967047:0:328777:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431529.967050:0:328777:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431529.967056:0:328777:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0e33cb8 00010000:00010000:2.0:1789431529.967060:0:328777:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:2.0:1789431529.967069:0:328777:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:2.0:1789431529.970649:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970666:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431529.970678:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970683:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.970691:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970694:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.970702:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970705:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.970713:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970716:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431529.970725:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970729:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.970737:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970740:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.970747:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970750:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431529.970759:0:328777:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431529.970763:0:328777:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431529.993063:0:328780:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431529.993081:0:328780:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431529.993089:0:328780:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431529.993115:0:328780:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431529.993133:0:328780:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431529.993148:0:328780:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:3.0:1789431529.993156:0:328780:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431529.993163:0:328780:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431529.993167:0:328780:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431529.993174:0:328780:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1563cb8 00010000:00010000:3.0:1789431529.993177:0:328780:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:3.0:1789431529.993187:0:328780:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1789431529.998594:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998612:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431529.998696:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998701:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.998711:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998715:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.998724:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998727:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.998737:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998740:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431529.998750:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998754:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.998763:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998767:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.998776:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998780:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431529.998788:0:328781:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431529.998791:0:328781:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431530.011530:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431530.011551:0:328785:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431530.011558:0:328785:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431530.011584:0:328785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431530.011604:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1789431530.011629:0:328785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:2.0:1789431530.011638:0:328785:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431530.011645:0:328785:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431530.011648:0:328785:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431530.011655:0:328785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1577cb8 00010000:00010000:2.0:1789431530.011766:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:2.0:1789431530.011776:0:328785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:2.0:1789431530.013004:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013016:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431530.013030:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013034:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431530.013043:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013046:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431530.013055:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013057:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431530.013064:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013067:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431530.013076:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013079:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431530.013086:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013089:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431530.013097:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013100:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431530.013109:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431530.013117:0:328785:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a92d80, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431530.013124:0:328785:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libsigsegv.so.2, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:2.0:1789431530.013143:0:328785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libsigsegv.so.2,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431530.013199:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431530.013209:0:328785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431530.013220:0:328785:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431530.013240:0:328785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2101180 x1876350913525120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431530.013250:0:328785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431530.013272:0:328785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525120:192.168.203.128@tcp:101:awk.0 00000080:00200000:1.0:1789431530.013739:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431530.013749:0:328784:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431530.013755:0:328784:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431530.013768:0:328784:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431530.013780:0:328784:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1789431530.013791:0:328784:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1789431530.013798:0:328784:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431530.013805:0:328784:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431530.013809:0:328784:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431530.013815:0:328784:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d9bcb8 00010000:00010000:1.0:1789431530.013819:0:328784:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1789431530.013828:0:328784:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1789431530.015027:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015037:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431530.015046:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015049:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431530.015056:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015059:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431530.015066:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015069:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431530.015076:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015079:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431530.015086:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015089:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431530.015097:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015100:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431530.015109:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015112:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431530.015119:0:328784:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431530.015123:0:328784:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431530.017400:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431530.017415:0:328783:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431530.017422:0:328783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431530.017446:0:328783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431530.019281:0:328783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431530.019306:0:328783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:3.0:1789431530.019500:0:328783:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431530.019512:0:328783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431530.019515:0:328783:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431530.019524:0:328783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c148bcb8 00010000:00010000:3.0:1789431530.019528:0:328783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:3.0:1789431530.019537:0:328783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000100:00100000:0.0:1789431530.019573:0:328785:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cecd80 going to sleep for 11 seconds 00000080:00200000:3.0:1789431530.022972:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.022987:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431530.023000:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023004:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431530.023014:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023017:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431530.023025:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023028:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431530.023036:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023039:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431530.023047:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023051:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431530.023061:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023064:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431530.023074:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023077:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431530.023085:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.023089:0:328783:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431530.024582:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431530.024594:0:328783:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431530.024601:0:328783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431530.024782:0:328783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431530.024801:0:328783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:3.0:1789431530.024816:0:328783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:3.0:1789431530.024824:0:328783:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431530.024901:0:328783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431530.024905:0:328783:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431530.024912:0:328783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c148bcb8 00010000:00010000:3.0:1789431530.024915:0:328783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:3.0:1789431530.024925:0:328783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:3.0:1789431530.024950:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.024954:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:3.0:1789431530.024959:0:328783:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:3.0:1789431530.024973:0:328783:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:3.0:1789431530.024984:0:328783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200008:3.0:1789431530.029438:0:328783:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431530.029452:0:328783:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431530.029474:0:328783:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431530.029503:0:328783:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431530.029519:0:328783:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431530.029528:0:328783:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431530.029534:0:328783:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431530.029537:0:328783:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:3.0:1789431530.029544:0:328783:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c148bcb8 00010000:00010000:3.0:1789431530.029547:0:328783:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431530.029556:0:328783:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431530.029586:0:328783:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431530.029593:0:328783:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:3.0:1789431530.029597:0:328783:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3888, max_age 3903 00800000:00000004:3.0:1789431530.029606:0:328783:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3888, max_age 3903 00000100:00100000:3.0:1789431530.029656:0:328783:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328783:1876350913525248:192.168.203.128@tcp:41:df.0 00000100:00100000:0.0:1789431530.031275:0:328785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525120:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431530.031296:0:328785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031317:0:328785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031324:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031332:0:328785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031340:0:328785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031346:0:328785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431530.031350:0:328785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031357:0:328785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031364:0:328785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031371:0:328785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431530.031376:0:328785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.031383:0:328785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea29 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431530.031399:0:328785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2101180 x1876350913525120/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431546 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431530.031417:0:328785:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431530.031439:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431530.031450:0:328785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00010000:0.0:1789431530.031458:0:328785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c15779a0 00010000:00010000:0.0:1789431530.031461:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1789431530.031469:0:328785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1789431530.032030:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032040:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.032050:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032054:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.032063:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032067:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.032077:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032080:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.032088:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032091:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.032099:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032102:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.032110:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032113:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.032121:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032123:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.032130:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.032140:0:328785:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811e0082c00, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431530.032146:0:328785:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libreadline.so.7, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431530.032165:0:328785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libreadline.so.7,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431530.032208:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.032216:0:328785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431530.032228:0:328785:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431530.032244:0:328785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913525504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431530.032253:0:328785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431530.032272:0:328785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525504:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431530.032587:0:328785:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:2.0:1789431530.032793:0:328783:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db4300 going to sleep for 11 seconds 00000100:00100000:1.0:1789431530.033595:0:328783:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328783:1876350913525248:192.168.203.128@tcp:41:df.0 00800000:00000004:1.0:1789431530.033625:0:328783:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:1.0:1789431530.033632:0:328783:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:1.0:1789431530.033637:0:328783:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000100:00100000:0.0:1789431530.040796:0:328785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525504:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431530.040818:0:328785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040916:0:328785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040924:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040932:0:328785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040941:0:328785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040947:0:328785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431530.040951:0:328785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040957:0:328785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040964:0:328785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040971:0:328785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431530.040975:0:328785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.040981:0:328785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27ea30 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431530.040999:0:328785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913525504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431546 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431530.041013:0:328785:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431530.041039:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1789431530.041057:0:328785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00010000:0.0:1789431530.041066:0:328785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c15779a0 00010000:00010000:0.0:1789431530.041070:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1789431530.041079:0:328785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1789431530.041568:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041578:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.041591:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041595:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.041604:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041606:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.041718:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041724:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.041737:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041740:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.041748:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041752:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.041760:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041763:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.041771:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041774:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.041780:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.041790:0:328785:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811ca227080, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431530.041796:0:328785:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libmpfr.so.4, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431530.041812:0:328785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libmpfr.so.4,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431530.042064:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.042074:0:328785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431530.042084:0:328785:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431530.042091:0:328785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913525760/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431530.042101:0:328785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431530.042120:0:328785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525760:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431530.042320:0:328785:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431530.047038:0:328785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913525760:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431530.047061:0:328785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047074:0:328785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047138:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047146:0:328785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047155:0:328785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047165:0:328785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431530.047170:0:328785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047178:0:328785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047185:0:328785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047258:0:328785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431530.047265:0:328785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.047271:0:328785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3400/0x1ffb2ee1bd27ea37 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431530.047290:0:328785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913525760/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431546 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431530.047303:0:328785:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431530.047328:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1789431530.047339:0:328785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00010000:0.0:1789431530.047348:0:328785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c15779a0 00010000:00010000:0.0:1789431530.047353:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1789431530.047365:0:328785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:0.0:1789431530.048298:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048309:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.048322:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048326:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.048335:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048338:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.048347:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048350:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.048358:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048362:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.048370:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048373:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.048381:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048385:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.048394:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048397:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.048404:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.048415:0:328785:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c3985c80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431530.048421:0:328785:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libgmp.so.10, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431530.048441:0:328785:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libgmp.so.10,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431530.048488:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.048497:0:328785:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431530.048508:0:328785:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431530.048516:0:328785:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913526016/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431530.048525:0:328785:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431530.048545:0:328785:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913526016:192.168.203.128@tcp:101:awk.0 00000100:00100000:0.0:1789431530.049206:0:328785:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a28480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431530.054456:0:328785:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job awk:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:328785:1876350913526016:192.168.203.128@tcp:101:awk.0 00010000:00010000:0.0:1789431530.054478:0:328785:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054490:0:328785:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054498:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054506:0:328785:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054514:0:328785:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054520:0:328785:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c3600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431530.054524:0:328785:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054536:0:328785:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054543:0:328785:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054550:0:328785:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431530.054555:0:328785:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431530.054561:0:328785:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3600/0x1ffb2ee1bd27ea3e lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 328785 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431530.054590:0:328785:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd7800 x1876350913526016/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431546 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'awk.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431530.054603:0:328785:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431530.054634:0:328785:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1789431530.054645:0:328785:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00010000:0.0:1789431530.054652:0:328785:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c15779a0 00010000:00010000:0.0:1789431530.054656:0:328785:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1789431530.054663:0:328785:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1789431530.055035:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055043:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.055053:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055056:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055064:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055066:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055072:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055076:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055083:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055085:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055093:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055097:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055105:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055108:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055116:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055118:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055125:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055128:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431530.055262:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055267:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.055275:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055277:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055284:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055286:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055293:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055295:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055302:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055305:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055311:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055314:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055321:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055324:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055330:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055332:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055339:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055342:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:0.0:1789431530.055449:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055454:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.055462:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055464:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055470:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055473:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055482:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055485:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055492:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055494:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055503:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055506:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055514:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055517:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055524:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055527:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055535:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055539:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431530.055699:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055704:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431530.055711:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055714:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055720:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055723:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055730:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055732:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055740:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055742:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431530.055749:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055751:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055758:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055760:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055767:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055769:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431530.055776:0:328785:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431530.055779:0:328785:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00010000:00010000:3.0:1789431530.149976:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431531.183989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431531.306790:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913526144:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431531.307059:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913526400:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431531.307165:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789431531.307173:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913526272:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431531.313955:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913526272:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431531.313972:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913526144:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431531.313986:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc6300 x1876350913526400/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431547 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431531.313996:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913526400:192.168.203.128@tcp:400:kworker.0 00000001:02000400:1.0:1789431531.356422:0:328918:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:1.0:1789431532.198019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431532.853053:0:329057:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:3.0:1789431533.223926:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431534.246852:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431535.270348:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431536.303332:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431536.428424:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913526528:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431536.428538:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913526784:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431536.428574:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431536.428581:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913526656:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431536.435591:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913526656:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431536.435647:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2103b80 x1876350913526784/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431552 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431536.435659:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913526784:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431536.435724:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913526528:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431537.318732:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431538.342936:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431539.377271:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431539.423341:0:329635:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_10: symlink returns EROFS if the file name2 would reside on a read-only file system ========================================================== 20:18:58 (1789431538) 00010000:00010000:3.0:1789431540.390001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431541.086486:0:329773:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/10.t against ext4 filesystem 00010000:00010000:2.0:1789431541.421184:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431541.542427:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913526912:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431541.542532:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913527168:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431541.542569:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789431541.542574:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913527040:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431541.551426:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913527040:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431541.551443:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913526912:192.168.203.128@tcp:400:kworker.0 00010000:00080000:2.0:1789431541.551455:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2103480 x1876350913527168/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431557 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431541.551467:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913527168:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431542.438433:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431543.359790:0:329924:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/10.t against lustre filesystem 00010000:00010000:3.0:1789431543.461985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1789431543.834649:0:329936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431543.834682:0:329936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431543.834701:0:329936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431543.834717:0:329936:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431543.834725:0:329936:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431543.834733:0:329936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431543.834737:0:329936:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431543.834744:0:329936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1503b58 00010000:00010000:0.0:1789431543.834748:0:329936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431543.834757:0:329936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431543.834779:0:329936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431543.834787:0:329936:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=13 00000080:00200008:0.0:1789431543.834796:0:329936:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1789431543.834801:0:329936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431543.834814:0:329936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431543.834821:0:329936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1789431543.834829:0:329936:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1789431543.835390:0:329936:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431543.835397:0:329936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431543.835399:0:329936:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431543.835404:0:329936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1503cb8 00010000:00010000:0.0:1789431543.835408:0:329936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1789431543.835624:0:329936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200008:0.0:1789431543.836251:0:329936:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1789431543.836257:0:329936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431543.836273:0:329936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431543.836286:0:329936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431543.836302:0:329936:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431543.836309:0:329936:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431543.836314:0:329936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431543.836317:0:329936:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431543.836322:0:329936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1503cb8 00010000:00010000:0.0:1789431543.836325:0:329936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431543.836333:0:329936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1789431543.838113:0:329936:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431543.838151:0:329936:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431543.838167:0:329936:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431543.838181:0:329936:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1789431543.838189:0:329936:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431543.838195:0:329936:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431543.838199:0:329936:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27e9ce 00000080:00010000:0.0:1789431543.838204:0:329936:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e9ce from it ffffbc42c1503bb0 00010000:00010000:0.0:1789431543.838208:0:329936:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431543.838218:0:329936:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:0.0:1789431543.838232:0:329936:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:0.0:1789431543.838240:0:329936:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:0.0:1789431543.838259:0:329936:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7fff8f7591c0 00000080:00200000:0.0:1789431543.838273:0:329936:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:0.0:1789431543.838299:0:329936:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1789431543.838348:0:329936:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1789431543.838356:0:329936:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431543.838359:0:329936:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2a00/0x1ffb2ee1bd27e9ce lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad1cb expref: -99 pid: 328338 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:0.0:1789431543.838382:0:329936:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431543.838396:0:329936:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913527424/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431543.838434:0:329936:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329936:1876350913527424:192.168.203.128@tcp:36:lfs.0 00000100:00100000:0.0:1789431543.840326:0:329936:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db4840 going to sleep for 11 seconds 00000100:00100000:0.0:1789431543.850223:0:329936:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329936:1876350913527424:192.168.203.128@tcp:36:lfs.0 00000080:00200008:0.0:1789431543.850332:0:329936:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200008:3.0:1789431543.978489:0:329503:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431543.978500:0:329503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431543.978523:0:329503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431543.978583:0:329503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 329503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.978593:0:329503:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 329503 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.978604:0:329503:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 329503 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431543.978628:0:329503:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329503:1876350913527552:192.168.203.128@tcp:101:bash.0 00000100:00100000:3.0:1789431543.978716:0:329503:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431543.985602:0:329503:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329503:1876350913527552:192.168.203.128@tcp:101:bash.0 00010000:00010000:3.0:1789431543.985626:0:329503:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985640:0:329503:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985657:0:329503:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985667:0:329503:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985678:0:329503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985686:0:329503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985694:0:329503:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431543.985704:0:329503:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc6300 x1876350913527552/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431559 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431543.985717:0:329503:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985729:0:329503:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1789431543.985734:0:329503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431543.985758:0:329503:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431543.985764:0:329503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:3.0:1789431543.985769:0:329503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431543.985772:0:329503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c93cb8 00010000:00010000:3.0:1789431543.985775:0:329503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431543.985795:0:329503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431543.985868:0:329503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431543.985882:0:329503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431543.985890:0:329503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431543.985900:0:329503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431543.985906:0:329503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431543.985910:0:329503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431543.986219:0:329503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:3.0:1789431543.986228:0:329503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c93b58 00010000:00010000:3.0:1789431543.986231:0:329503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431543.986239:0:329503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431543.986248:0:329503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.986255:0:329503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431543.986264:0:329503:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431543.986268:0:329503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431543.986282:0:329503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431543.986291:0:329503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:3.0:1789431543.986300:0:329503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:3.0:1789431543.986306:0:329503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431543.986310:0:329503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431543.986312:0:329503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431543.986316:0:329503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c93cb8 00010000:00010000:3.0:1789431543.986319:0:329503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:3.0:1789431543.986326:0:329503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:3.0:1789431543.986347:0:329503:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431543.986354:0:329503:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431543.986360:0:329503:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431543.986368:0:329503:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431543.986375:0:329503:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431543.986378:0:329503:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431543.986380:0:329503:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:3.0:1789431543.986384:0:329503:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c93c18 00010000:00010000:3.0:1789431543.986387:0:329503:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431543.986393:0:329503:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431543.986400:0:329503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.986404:0:329503:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:3.0:1789431543.986409:0:329503:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:2.0:1789431543.992950:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.992966:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431543.993087:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993093:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993103:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993107:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993116:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993119:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993127:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993130:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993149:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993153:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.993162:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993165:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.993173:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993176:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.993184:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993188:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:2.0:1789431543.993406:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993411:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431543.993420:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993423:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993430:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993433:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993440:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993443:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.993451:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.993453:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.995231:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.995244:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.995265:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.995269:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.995278:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.995281:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.995289:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.995292:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431543.996728:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996740:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431543.996756:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996760:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.996769:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996772:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.996797:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996800:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.996809:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996812:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.996820:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.996823:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997070:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997076:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997090:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997094:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997102:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997106:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:2.0:1789431543.997338:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997343:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431543.997354:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997357:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997366:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997369:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997377:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997380:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997388:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997391:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997400:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997404:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997411:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997414:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997423:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997427:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997435:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997439:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431543.997561:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997567:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431543.997575:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997578:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997586:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997589:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997598:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997601:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997609:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997612:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431543.997620:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997623:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997631:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997634:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997642:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997645:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431543.997654:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431543.997658:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431543.999468:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999491:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431543.999511:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999516:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999529:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999532:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999540:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999544:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999552:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999555:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999563:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999566:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999577:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999580:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999589:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999592:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999601:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999605:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:3.0:1789431543.999822:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999827:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431543.999858:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999862:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999871:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999873:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999880:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999882:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999890:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999892:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431543.999900:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999903:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999909:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999912:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999918:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999921:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431543.999928:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431543.999933:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:3.0:1789431544.005358:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005376:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431544.005392:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005397:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.005406:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005409:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.005420:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005423:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.005431:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005435:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.005444:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005447:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.005455:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005458:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.005466:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005470:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.005478:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.005482:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431544.270681:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270699:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431544.270712:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270715:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431544.270725:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270728:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431544.270737:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270739:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431544.270748:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270751:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431544.270761:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270764:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431544.270772:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270775:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431544.270782:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270785:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431544.270793:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431544.270797:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=librt.so.1, flags=257 00000080:00200000:1.0:1789431544.305502:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.305518:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.proverc, flags=1 00000080:00200000:1.0:1789431544.307711:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.307727:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.prove, flags=1 00000080:00200000:1.0:1789431544.338146:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.338162:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.338169:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=testrules.yml, flags=1 00000080:00200000:1.0:1789431544.338182:0:329939:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.338186:0:329939:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t, flags=17 00010000:00010000:3.0:1789431544.486085:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789431544.593721:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593737:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.593751:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593755:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.593764:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593768:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.593776:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593780:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.593788:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593791:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.593799:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593802:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.593809:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593812:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.593819:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593822:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.593829:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.593856:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:2.0:1789431544.594028:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594033:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.594040:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594043:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.594050:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594052:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.594059:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594062:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.594068:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594071:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.594077:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594080:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.594086:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594088:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.594094:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594097:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.594103:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.594106:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431544.596973:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.596988:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.597000:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597005:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.597014:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597018:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.597025:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597027:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.597035:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597038:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.597046:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597049:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.597058:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597061:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.597069:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597072:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.597080:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.597083:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:2.0:1789431544.598359:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598371:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.598383:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598388:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598395:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598399:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598409:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598412:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598421:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598424:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598432:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598435:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598444:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598447:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598455:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598458:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598466:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598470:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431544.598595:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598600:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.598609:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598612:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598619:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598622:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598629:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598632:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598639:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598641:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.598648:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598651:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598661:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598663:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598671:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598674:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.598682:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.598685:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431544.600674:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600688:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.600701:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600704:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.600712:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600729:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.600740:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600743:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.600751:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600753:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.600760:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600764:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.600772:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600776:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.600785:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600788:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.600796:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600800:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:2.0:1789431544.600971:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600976:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.600985:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.600989:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.600997:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601000:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601008:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601011:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601019:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601022:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601031:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601034:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601042:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601046:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601054:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601058:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601066:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601070:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:2.0:1789431544.601227:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601232:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.601241:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601244:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601252:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601255:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601265:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601267:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601275:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601278:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.601284:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601287:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601294:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601297:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601306:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601308:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.601315:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.601319:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431544.610342:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.610360:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.610374:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.610378:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.610386:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.610390:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.610398:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611502:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.611534:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611539:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.611550:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611553:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.611561:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611564:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.611572:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611575:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.611583:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.611587:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:2.0:1789431544.612670:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.612681:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.612692:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.612696:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.612704:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.612707:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613434:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613442:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613457:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613460:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613469:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613472:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613481:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613484:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613491:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613494:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613501:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613505:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431544.613671:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613677:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431544.613687:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613691:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613699:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613702:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613709:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613711:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613733:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613737:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431544.613745:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613749:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613758:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613761:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613770:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613772:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431544.613780:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431544.613783:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431544.618366:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431544.618393:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.618783:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431544.618799:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431544.618807:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.618815:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431544.618819:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:1.0:1789431544.618826:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1cabb58 00010000:00010000:1.0:1789431544.618860:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431544.618870:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431544.618887:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.618895:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431544.618906:0:329940:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431544.618911:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431544.618925:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.618944:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1789431544.618952:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:1.0:1789431544.618959:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.618963:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431544.618966:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431544.618969:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1cabcb8 00010000:00010000:1.0:1789431544.618972:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:1.0:1789431544.618978:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:1.0:1789431544.618995:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431544.619000:0:329940:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431544.619003:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431544.619009:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.619014:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1789431544.619022:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:1.0:1789431544.619027:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.619030:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431544.619033:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431544.619036:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1cabcb8 00010000:00010000:1.0:1789431544.619039:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:1.0:1789431544.619045:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200008:1.0:1789431544.619067:0:329940:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431544.619071:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431544.619077:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.619083:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431544.619090:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431544.619096:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.619100:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431544.619102:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:1.0:1789431544.619106:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1cabcb8 00010000:00010000:1.0:1789431544.619109:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431544.619116:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431544.619141:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431544.619148:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.619153:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431544.619162:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431544.619168:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.619172:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431544.619174:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:1.0:1789431544.619177:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1cabb58 00010000:00010000:1.0:1789431544.619180:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431544.619188:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431544.619196:0:329940:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.619200:0:329940:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431544.619205:0:329940:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431544.619209:0:329940:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431544.619217:0:329940:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431544.619223:0:329940:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1789431544.619231:0:329940:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:1.0:1789431544.619236:0:329940:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431544.619239:0:329940:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431544.619242:0:329940:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431544.619245:0:329940:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1cabcb8 00010000:00010000:1.0:1789431544.619248:0:329940:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:1.0:1789431544.619255:0:329940:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:3.0:1789431544.621799:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431544.621819:0:329941:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431544.621826:0:329941:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431544.621876:0:329941:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431544.621894:0:329941:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:3.0:1789431544.621908:0:329941:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:3.0:1789431544.621916:0:329941:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431544.621923:0:329941:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431544.621926:0:329941:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431544.621933:0:329941:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:3.0:1789431544.621937:0:329941:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:3.0:1789431544.621945:0:329941:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:3.0:1789431544.626626:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626641:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431544.626653:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626658:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.626668:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626671:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.626679:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626682:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.626691:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626694:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.626703:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626706:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.626713:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626716:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.626725:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626728:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.626735:0:329941:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.626738:0:329941:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431544.633512:0:329944:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431544.633528:0:329944:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431544.633536:0:329944:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431544.633562:0:329944:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431544.633580:0:329944:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431544.633595:0:329944:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:3.0:1789431544.633604:0:329944:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431544.633610:0:329944:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431544.633614:0:329944:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431544.633621:0:329944:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1593cb8 00010000:00010000:3.0:1789431544.633624:0:329944:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:3.0:1789431544.633633:0:329944:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:1.0:1789431544.636012:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636028:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.636040:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636044:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.636052:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636054:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.636061:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636064:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.636070:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636073:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.636079:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636082:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.636090:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636093:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.636100:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636103:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.636110:0:329945:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.636114:0:329945:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431544.646403:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431544.646426:0:329947:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431544.646433:0:329947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431544.646458:0:329947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431544.646476:0:329947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:3.0:1789431544.646491:0:329947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00000080:00200000:3.0:1789431544.646500:0:329947:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431544.646507:0:329947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431544.646511:0:329947:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431544.646519:0:329947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:3.0:1789431544.646522:0:329947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 43 lru_type: 1 00010000:00010000:3.0:1789431544.646531:0:329947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:3.0:1789431544.647810:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647822:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431544.647861:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647866:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.647875:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647878:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.647886:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647888:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.647895:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647897:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431544.647905:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647907:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.647915:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647919:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.647927:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647929:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431544.647936:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.647939:0:329947:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431544.648770:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431544.648780:0:329947:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431544.648787:0:329947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431544.648810:0:329947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431544.648828:0:329947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:3.0:1789431544.648868:0:329947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00000080:00200000:3.0:1789431544.648876:0:329947:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431544.648883:0:329947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431544.648887:0:329947:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431544.648893:0:329947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:3.0:1789431544.648898:0:329947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 44 lru_type: 1 00010000:00010000:3.0:1789431544.648908:0:329947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:3.0:1789431544.648931:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.648935:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:3.0:1789431544.648940:0:329947:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:3.0:1789431544.648954:0:329947:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:3.0:1789431544.648982:0:329947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:0.0:1789431544.649725:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431544.649734:0:329948:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431544.649739:0:329948:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431544.649752:0:329948:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431544.649763:0:329948:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:0.0:1789431544.649774:0:329948:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00000080:00200000:0.0:1789431544.649780:0:329948:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431544.649785:0:329948:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431544.649788:0:329948:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431544.649793:0:329948:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c163fcb8 00010000:00010000:0.0:1789431544.649795:0:329948:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 45 lru_type: 1 00010000:00010000:0.0:1789431544.649802:0:329948:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:1.0:1789431544.652726:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652742:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.652754:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652758:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.652765:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652769:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.652775:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652778:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.652785:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652788:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.652795:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652798:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.652806:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652809:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.652817:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652820:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.652828:0:329948:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.652856:0:329948:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431544.654165:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431544.654178:0:329949:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431544.654185:0:329949:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431544.654209:0:329949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431544.654227:0:329949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1789431544.654242:0:329949:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00000080:00200000:2.0:1789431544.654251:0:329949:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431544.654257:0:329949:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431544.654260:0:329949:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431544.654267:0:329949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1677cb8 00010000:00010000:2.0:1789431544.654270:0:329949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 46 lru_type: 1 00010000:00010000:2.0:1789431544.654278:0:329949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200008:3.0:1789431544.657543:0:329947:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431544.657554:0:329947:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431544.657574:0:329947:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431544.657591:0:329947:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431544.657605:0:329947:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431544.657612:0:329947:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431544.657618:0:329947:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431544.657622:0:329947:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:3.0:1789431544.657628:0:329947:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c0da3cb8 00010000:00010000:3.0:1789431544.657632:0:329947:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431544.657656:0:329947:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431544.657683:0:329947:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431544.657690:0:329947:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:3.0:1789431544.657693:0:329947:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3904, max_age 3918 00800000:00000004:3.0:1789431544.657703:0:329947:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3904, max_age 3918 00000100:00100000:3.0:1789431544.657743:0:329947:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329947:1876350913527680:192.168.203.128@tcp:41:df.0 00000100:00100000:3.0:1789431544.661387:0:329947:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8525f00 going to sleep for 11 seconds 00000080:00200000:1.0:1789431544.662119:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662140:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.662154:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662159:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662167:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662171:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662178:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662182:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662189:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662192:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662200:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662204:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662211:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662214:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662220:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662223:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662231:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662235:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431544.662411:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662416:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.662424:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662427:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662435:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662438:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662447:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662450:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662457:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662459:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662468:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662470:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662478:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662481:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662488:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662491:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662500:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662503:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431544.662635:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662641:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.662651:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662654:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662661:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662664:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662672:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662675:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662683:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662685:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662693:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662696:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662703:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662706:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662714:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662716:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662724:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662728:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431544.662876:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662881:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.662890:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662893:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662900:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662903:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662909:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662912:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662920:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662922:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.662931:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662934:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662943:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662946:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662953:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662957:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.662964:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.662968:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431544.663093:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663098:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.663107:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663109:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663116:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663119:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663126:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663128:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663151:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663154:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663162:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663165:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663174:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663177:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663187:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663189:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663198:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663202:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431544.663321:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663326:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.663333:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663335:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663343:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663346:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663354:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663357:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663364:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663367:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663376:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663379:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663390:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663393:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663402:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663405:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663412:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663416:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431544.663534:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663539:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.663548:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663550:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663558:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663560:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663567:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663570:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663577:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663580:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663588:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663591:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663598:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663601:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663608:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663611:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663618:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663622:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431544.663784:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663790:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431544.663798:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663801:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663808:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663811:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663819:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663822:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663830:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663854:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431544.663865:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663868:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663877:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663880:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663886:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663889:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431544.663896:0:329949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431544.663899:0:329949:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000100:00100000:3.0:1789431544.666803:0:329947:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:329947:1876350913527680:192.168.203.128@tcp:41:df.0 00800000:00000004:3.0:1789431544.666853:0:329947:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431544.666861:0:329947:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431544.666866:0:329947:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00010000:00010000:1.0:1789431545.524428:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431546.197667:0:330082:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:0.0:1789431546.534715:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431546.666887:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913527808:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431546.667198:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913528064:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431546.667246:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789431546.667252:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913527936:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431546.677207:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913527936:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431546.677263:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913527808:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431546.677287:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc4a80 x1876350913528064/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431562 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431546.677308:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913528064:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431547.558247:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431547.856737:0:330221:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:1.0:1789431548.583111:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431549.606800:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431550.630169:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431551.654940:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789431551.783704:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913528192:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431551.783814:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913528448:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431551.783868:0:2389:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789431551.783875:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913528320:192.168.203.128@tcp:400:kworker.0 00010000:00080000:1.0:1789431551.801451:0:2389:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d0bdce00 x1876350913528448/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431567 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431551.801472:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913528448:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431551.801913:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913528320:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431551.801940:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913528192:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431552.680362:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431553.720218:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431554.727988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1789431555.521710:0:330800:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_11: symlink returns ENOSPC if there are no free inodes on the file system on which the symbolic link is being created ========================================================== 20:19:14 (1789431554) 00010000:00010000:3.0:1789431555.750034:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431556.775094:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431556.909016:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913528576:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431556.909117:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913528832:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431556.909403:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913528704:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431556.916654:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913528576:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431556.932150:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913528704:192.168.203.128@tcp:400:kworker.0 00010000:00080000:3.0:1789431556.932198:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc4700 x1876350913528832/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431572 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431556.932210:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913528832:192.168.203.128@tcp:400:kworker.0 00000001:02000400:2.0:1789431557.136018:0:330938:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/11.t against ext4 filesystem 00010000:00010000:0.0:1789431557.798111:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431558.823334:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431559.481912:0:331089:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/11.t against lustre filesystem 00000080:00200000:0.0:1789431559.783494:0:331102:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431559.783525:0:331102:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431559.783542:0:331102:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431559.783559:0:331102:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431559.783567:0:331102:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431559.783574:0:331102:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431559.783578:0:331102:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:0.0:1789431559.783586:0:331102:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c0bb58 00010000:00010000:0.0:1789431559.783590:0:331102:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:0.0:1789431559.783598:0:331102:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431559.783613:0:331102:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.783621:0:331102:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=13 00000080:00200008:0.0:1789431559.783631:0:331102:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1789431559.783636:0:331102:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431559.783648:0:331102:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431559.783657:0:331102:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:0.0:1789431559.783665:0:331102:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00000080:00200000:0.0:1789431559.783671:0:331102:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431559.783675:0:331102:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431559.783678:0:331102:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431559.783682:0:331102:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c0bcb8 00010000:00010000:0.0:1789431559.783685:0:331102:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 47 lru_type: 1 00010000:00010000:0.0:1789431559.783693:0:331102:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200008:0.0:1789431559.785434:0:331102:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1789431559.785443:0:331102:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431559.785462:0:331102:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431559.785479:0:331102:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431559.785493:0:331102:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431559.785501:0:331102:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431559.785507:0:331102:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431559.785510:0:331102:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:0.0:1789431559.785516:0:331102:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c0bcb8 00010000:00010000:0.0:1789431559.785520:0:331102:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431559.785529:0:331102:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1789431559.785654:0:331102:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431559.785662:0:331102:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431559.785668:0:331102:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431559.785677:0:331102:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:0.0:1789431559.785689:0:331102:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431559.785693:0:331102:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431559.785696:0:331102:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea45 00000080:00010000:0.0:1789431559.785700:0:331102:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea45 from it ffffbc42c1c0bbb0 00010000:00010000:0.0:1789431559.785703:0:331102:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431559.785711:0:331102:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:0.0:1789431559.785719:0:331102:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:0.0:1789431559.785726:0:331102:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:0.0:1789431559.785742:0:331102:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffcc42e8c90 00000080:00200000:0.0:1789431559.785753:0:331102:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:0.0:1789431559.785774:0:331102:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1789431559.787162:0:331102:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1789431559.787171:0:331102:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431559.787174:0:331102:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea45 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad329 expref: -99 pid: 329503 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:0.0:1789431559.787201:0:331102:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431559.787212:0:331102:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd7800 x1876350913529088/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431559.787235:0:331102:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331102:1876350913529088:192.168.203.128@tcp:36:lfs.0 00000100:00100000:0.0:1789431559.787332:0:331102:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c4506a80 going to sleep for 11 seconds 00000100:00100000:1.0:1789431559.794540:0:331102:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331102:1876350913529088:192.168.203.128@tcp:36:lfs.0 00000080:00200008:1.0:1789431559.794634:0:331102:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00010000:00010000:2.0:1789431559.847686:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1789431559.941677:0:330668:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431559.941804:0:330668:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431559.942188:0:330668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431559.942246:0:330668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 330668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.942257:0:330668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 330668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.942267:0:330668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 330668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431559.942291:0:330668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:330668:1876350913529216:192.168.203.128@tcp:101:bash.0 00000100:00100000:3.0:1789431559.942376:0:330668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db46c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431559.957881:0:330668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:330668:1876350913529216:192.168.203.128@tcp:101:bash.0 00010000:00010000:3.0:1789431559.957907:0:330668:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957918:0:330668:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957935:0:330668:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957947:0:330668:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957958:0:330668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957966:0:330668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.957973:0:330668:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431559.957985:0:330668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4700 x1876350913529216/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431575 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431559.957999:0:330668:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.958015:0:330668:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1789431559.958022:0:330668:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431559.958181:0:330668:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431559.958190:0:330668:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:3.0:1789431559.958196:0:330668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431559.958199:0:330668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1d73cb8 00010000:00010000:3.0:1789431559.958203:0:330668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431559.958212:0:330668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431559.958267:0:330668:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431559.958288:0:330668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431559.958300:0:330668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431559.958313:0:330668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431559.958320:0:330668:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431559.958324:0:330668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431559.958326:0:330668:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:3.0:1789431559.958338:0:330668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1d73b58 00010000:00010000:3.0:1789431559.958341:0:330668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431559.958352:0:330668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431559.958360:0:330668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431559.958367:0:330668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431559.958375:0:330668:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431559.958380:0:330668:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431559.958394:0:330668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431559.958405:0:330668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:3.0:1789431559.958415:0:330668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00000080:00200000:3.0:1789431559.958423:0:330668:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431559.958427:0:330668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431559.958430:0:330668:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431559.958434:0:330668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d73cb8 00010000:00010000:3.0:1789431559.958437:0:330668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 48 lru_type: 1 00010000:00010000:3.0:1789431559.958444:0:330668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:3.0:1789431559.958462:0:330668:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431559.958468:0:330668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431559.958474:0:330668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431559.958482:0:330668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431559.958488:0:330668:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431559.958492:0:330668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431559.958494:0:330668:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:3.0:1789431559.958498:0:330668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1d73c18 00010000:00010000:3.0:1789431559.958501:0:330668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431559.958508:0:330668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431559.958515:0:330668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431559.958519:0:330668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:3.0:1789431559.958524:0:330668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:0.0:1789431559.967626:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.967642:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.967651:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.967655:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.967662:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.967665:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.967671:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.967674:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.969218:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.969231:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.969254:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.969609:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.969629:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.969634:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.969644:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.969648:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.969657:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.969661:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:0.0:1789431559.970545:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970553:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.970565:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970569:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970578:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970581:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970588:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970591:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970599:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970602:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970610:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970612:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970619:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970622:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970631:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970634:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970641:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970644:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431559.970795:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970800:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.970811:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970815:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970824:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970827:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970856:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970860:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970869:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970873:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.970883:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970901:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970915:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970918:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970927:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970930:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.970938:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.970941:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:0.0:1789431559.971484:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971491:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.971502:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971506:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971515:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971518:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971526:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971529:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971536:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971538:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971547:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971549:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971557:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971560:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971567:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971570:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971578:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971582:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431559.971722:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971728:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.971739:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971742:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971752:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971755:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971764:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971768:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971777:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971781:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.971790:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971794:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971802:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971806:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971815:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971818:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.971827:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.971869:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:0.0:1789431559.972176:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972183:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.972192:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972196:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972202:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972205:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972211:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972213:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972219:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972222:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972228:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972231:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972238:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972241:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972249:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972252:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972261:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972265:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:0.0:1789431559.972577:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972583:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.972591:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972593:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972600:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972602:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972610:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972612:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972619:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972622:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.972630:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972633:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972641:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972644:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972650:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972653:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.972659:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.972663:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:0.0:1789431559.974301:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974314:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431559.974345:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974350:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.974360:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974363:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.974372:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974375:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.974384:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974386:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431559.974394:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974398:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.974406:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974409:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.974417:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974420:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431559.974427:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431559.974430:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431560.271686:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.271702:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.273017:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273037:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.273049:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273053:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.273061:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273064:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.273073:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273076:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.273084:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273087:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.273096:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273098:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.273108:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273111:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.273118:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.273122:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=librt.so.1, flags=257 00000080:00200000:0.0:1789431560.313985:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.314001:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.proverc, flags=1 00000080:00200000:0.0:1789431560.315618:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.315696:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.prove, flags=1 00000080:00200000:0.0:1789431560.350300:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.350314:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.350322:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=testrules.yml, flags=1 00000080:00200000:0.0:1789431560.350334:0:331105:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.350337:0:331105:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t, flags=17 00000080:00200000:0.0:1789431560.615174:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615323:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.615342:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615346:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615354:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615357:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615363:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615366:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615374:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615376:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615383:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615387:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615394:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615396:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615403:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615405:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615414:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615419:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:0.0:1789431560.615752:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615759:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.615770:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615773:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615782:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615784:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615794:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615797:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615805:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615809:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.615819:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615822:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615855:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615860:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615873:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615876:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.615886:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.615890:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431560.616036:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616041:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.616052:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616055:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616063:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616066:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616075:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616077:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616085:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616088:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616095:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616098:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616105:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616108:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616118:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616121:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616129:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616133:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:0.0:1789431560.616692:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616699:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.616709:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616712:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616720:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616723:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616730:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616733:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616741:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616744:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616752:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616755:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616763:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616765:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616774:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616777:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.616783:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616787:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431560.616973:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616978:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.616985:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616988:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.616996:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.616999:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617006:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617009:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617018:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617032:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617041:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617045:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617053:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617055:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617062:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617065:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617072:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617076:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:0.0:1789431560.617383:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617388:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.617397:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617400:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617407:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617410:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617417:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617420:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617426:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617429:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617435:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617438:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617445:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617447:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617455:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617458:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617465:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617468:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:0.0:1789431560.617664:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617669:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.617677:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617680:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617688:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617690:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617697:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617700:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617707:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617709:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.617716:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617719:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617725:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617729:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617736:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617739:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.617746:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617749:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:0.0:1789431560.617990:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.617995:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.618005:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618008:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.618016:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618029:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.618039:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618042:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.618049:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618052:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.618059:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618062:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.618068:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618071:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.618078:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618081:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.618089:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.618092:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431560.628150:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628166:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.628175:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628179:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628267:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628273:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628287:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628290:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628300:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628302:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628310:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628313:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628320:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628323:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628330:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628332:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628340:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628343:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:0.0:1789431560.628505:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628522:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.628531:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628534:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628542:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628545:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628552:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628554:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628564:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628566:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628574:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628576:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628583:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628586:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628593:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628596:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628604:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628607:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431560.628705:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628709:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.628717:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628721:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628728:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628730:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628737:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628740:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628747:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628750:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.628756:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628759:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628768:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628770:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628778:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628780:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.628787:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.628790:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431560.634477:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431560.634506:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.634917:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431560.634934:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431560.634943:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.634951:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431560.634955:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:0.0:1789431560.634963:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c0da3b58 00010000:00010000:0.0:1789431560.634966:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431560.634981:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431560.634996:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.635003:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431560.635014:0:331106:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431560.635019:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431560.635043:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.635051:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:0.0:1789431560.635058:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00000080:00200000:0.0:1789431560.635064:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.635068:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431560.635071:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431560.635074:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431560.635077:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 49 lru_type: 1 00010000:00010000:0.0:1789431560.635085:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:0.0:1789431560.635101:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431560.635106:0:331106:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431560.635110:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431560.635116:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.635121:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:0.0:1789431560.635129:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00000080:00200000:0.0:1789431560.635135:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.635138:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431560.635140:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431560.635143:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431560.635146:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 50 lru_type: 1 00010000:00010000:0.0:1789431560.635154:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200008:0.0:1789431560.635173:0:331106:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431560.635176:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431560.635180:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.635185:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431560.635926:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1789431560.635934:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.635940:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431560.635942:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:0.0:1789431560.635947:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431560.635951:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789431560.635958:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431560.635983:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431560.635991:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.635999:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431560.636006:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1789431560.636012:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.636016:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431560.636018:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:0.0:1789431560.636031:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c0da3b58 00010000:00010000:0.0:1789431560.636034:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:0.0:1789431560.636041:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431560.636048:0:331106:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.636054:0:331106:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431560.636061:0:331106:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431560.636064:0:331106:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431560.636072:0:331106:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431560.636077:0:331106:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:0.0:1789431560.636084:0:331106:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00000080:00200000:0.0:1789431560.636090:0:331106:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431560.636094:0:331106:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431560.636096:0:331106:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431560.636099:0:331106:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431560.636102:0:331106:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 51 lru_type: 1 00010000:00010000:0.0:1789431560.636109:0:331106:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:2.0:1789431560.639389:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431560.639406:0:331107:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431560.639413:0:331107:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431560.639437:0:331107:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431560.639454:0:331107:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:2.0:1789431560.639468:0:331107:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00000080:00200000:2.0:1789431560.639476:0:331107:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431560.639483:0:331107:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431560.639486:0:331107:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431560.639492:0:331107:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c43cb8 00010000:00010000:2.0:1789431560.639496:0:331107:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 52 lru_type: 1 00010000:00010000:2.0:1789431560.639505:0:331107:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:2.0:1789431560.640684:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640696:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431560.640706:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640709:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.640717:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640720:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.640729:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640732:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.640738:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640741:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.640748:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640751:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.640758:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640760:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.640767:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640770:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.640778:0:331107:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.640781:0:331107:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431560.661983:0:331110:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431560.662000:0:331110:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431560.662007:0:331110:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431560.662044:0:331110:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431560.662062:0:331110:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:3.0:1789431560.662076:0:331110:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00000080:00200000:3.0:1789431560.662090:0:331110:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431560.662097:0:331110:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431560.662100:0:331110:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431560.662107:0:331110:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d33cb8 00010000:00010000:3.0:1789431560.662111:0:331110:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 53 lru_type: 1 00010000:00010000:3.0:1789431560.662120:0:331110:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:0.0:1789431560.665426:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665442:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431560.665454:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665458:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.665466:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665468:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.665477:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665480:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.665488:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665492:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431560.665501:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665504:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.665591:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665596:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.665605:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665608:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431560.665616:0:331111:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431560.665619:0:331111:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431560.677434:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431560.677454:0:331113:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431560.677461:0:331113:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431560.677485:0:331113:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431560.677502:0:331113:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:3.0:1789431560.677587:0:331113:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00000080:00200000:3.0:1789431560.677596:0:331113:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431560.677603:0:331113:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431560.677606:0:331113:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431560.677614:0:331113:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d9bcb8 00010000:00010000:3.0:1789431560.677617:0:331113:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 54 lru_type: 1 00010000:00010000:3.0:1789431560.677633:0:331113:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000080:00200000:1.0:1789431560.680710:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431560.680729:0:331115:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431560.680739:0:331115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431560.680767:0:331115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431560.680785:0:331115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1789431560.680802:0:331115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00000080:00200000:1.0:1789431560.680811:0:331115:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431560.680820:0:331115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431560.680824:0:331115:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431560.680847:0:331115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431560.680851:0:331115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 55 lru_type: 1 00010000:00010000:1.0:1789431560.680859:0:331115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:2.0:1789431560.681686:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431560.681696:0:331114:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431560.681701:0:331114:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431560.681715:0:331114:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431560.681726:0:331114:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:2.0:1789431560.681737:0:331114:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00000080:00200000:2.0:1789431560.681744:0:331114:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431560.681749:0:331114:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431560.681752:0:331114:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431560.681756:0:331114:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1503cb8 00010000:00010000:2.0:1789431560.681760:0:331114:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 56 lru_type: 1 00010000:00010000:2.0:1789431560.681768:0:331114:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200000:3.0:1789431560.686010:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686040:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431560.686054:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686059:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431560.686067:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686070:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431560.686077:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686080:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431560.686087:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686090:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431560.686097:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686100:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431560.686107:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686109:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431560.686116:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686119:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431560.686125:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.686129:0:331113:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431560.688065:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688078:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431560.688090:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688093:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.688101:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688103:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.688110:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688113:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.688120:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688123:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431560.688130:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688133:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.688140:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688143:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.688150:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688153:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431560.688160:0:331114:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.688162:0:331114:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431560.689209:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689221:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.689231:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689234:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689241:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689244:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689252:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689255:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689263:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689266:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689274:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689277:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.689285:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689287:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.689294:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689297:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.689304:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689308:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431560.689459:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689465:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.689474:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689478:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689486:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689489:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.689499:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.689501:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.691751:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.691770:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.691789:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.691793:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.691801:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.691804:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.691812:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.691815:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.691823:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.691827:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431560.692044:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692050:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.692060:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692063:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692071:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692074:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692081:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692084:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692092:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692094:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692102:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692104:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692112:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692116:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692123:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692126:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692134:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692138:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431560.692267:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692273:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.692284:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692287:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692295:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692298:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692306:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692309:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692318:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692322:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.692330:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692333:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692340:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692343:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692354:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692357:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.692365:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692369:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431560.692494:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.692500:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431560.692723:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431560.692732:0:331113:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431560.692739:0:331113:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431560.692765:0:331113:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431560.692784:0:331113:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:2.0:1789431560.692805:0:331113:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00000080:00200000:2.0:1789431560.692814:0:331113:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431560.692821:0:331113:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431560.692825:0:331113:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431560.692846:0:331113:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d9bcb8 00010000:00010000:2.0:1789431560.692851:0:331113:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 57 lru_type: 1 00010000:00010000:2.0:1789431560.692860:0:331113:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000080:00200000:2.0:1789431560.692886:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431560.692890:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:2.0:1789431560.692896:0:331113:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:2.0:1789431560.692912:0:331113:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:2.0:1789431560.692924:0:331113:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1789431560.693859:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693868:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.693889:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693893:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.693903:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693906:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.693913:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693916:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.693924:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693927:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.693935:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693938:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.693947:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693950:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.693959:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.693963:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431560.694157:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694163:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.694176:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694179:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694188:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694192:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694200:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694203:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694212:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694214:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694220:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694223:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694229:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694232:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694239:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694242:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694250:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694253:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431560.694376:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694381:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.694392:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694395:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694403:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694406:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694416:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694419:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694429:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694433:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694441:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694444:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694452:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694455:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694465:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694469:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694476:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694479:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431560.694730:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694736:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431560.694746:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694749:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694756:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694759:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694767:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694770:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694778:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694780:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431560.694789:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694792:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694800:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694802:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694809:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694811:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431560.694818:0:331115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431560.694822:0:331115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200008:3.0:1789431560.697722:0:331113:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431560.697732:0:331113:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431560.697754:0:331113:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431560.697772:0:331113:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431560.697787:0:331113:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431560.697797:0:331113:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431560.697804:0:331113:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431560.697807:0:331113:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:3.0:1789431560.697815:0:331113:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c0d9bcb8 00010000:00010000:3.0:1789431560.697818:0:331113:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431560.697826:0:331113:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431560.697885:0:331113:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431560.697892:0:331113:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:3.0:1789431560.697896:0:331113:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3919, max_age 3934 00800000:00000004:3.0:1789431560.697908:0:331113:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3919, max_age 3934 00000100:00100000:3.0:1789431560.697949:0:331113:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331113:1876350913529344:192.168.203.128@tcp:41:df.0 00000100:00100000:3.0:1789431560.698048:0:331113:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8db4300 going to sleep for 11 seconds 00000100:00100000:3.0:1789431560.704326:0:331113:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331113:1876350913529344:192.168.203.128@tcp:41:df.0 00800000:00000004:3.0:1789431560.704356:0:331113:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431560.704374:0:331113:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431560.704379:0:331113:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00010000:00010000:1.0:1789431560.873442:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431561.899983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431561.986310:0:331247:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00000100:00100000:1.0:1789431562.025442:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913529472:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431562.025544:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913529728:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431562.025579:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789431562.025586:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913529600:192.168.203.128@tcp:400:kworker.0 00010000:00080000:1.0:1789431562.032619:0:2390:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d0bdea00 x1876350913529728/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431578 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431562.032639:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913529728:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431562.045522:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913529600:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431562.045541:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913529472:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431562.929376:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431563.544656:0:331386:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:1.0:1789431563.945923:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789431564.973984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431565.991635:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431567.014248:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431567.143125:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913529856:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431567.143267:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913530112:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431567.143301:0:2389:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789431567.143307:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913529984:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431567.156520:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913529856:192.168.203.128@tcp:400:kworker.0 00010000:00080000:0.0:1789431567.156544:0:2389:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811c8cd5880 x1876350913530112/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431583 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431567.156556:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913530112:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431567.162726:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913529984:192.168.203.128@tcp:400:kworker.0 00010000:00010000:1.0:1789431568.039173:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431569.069589:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431570.087030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1789431570.087517:0:331963:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test symlink_12: symlink returns EFAULT if one of the pathnames specified is outside the process's allocated address space ========================================================== 20:19:28 (1789431568) 00010000:00010000:1.0:1789431571.110318:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431571.643065:0:332101:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/12.t against ext4 filesystem 00010000:00010000:3.0:1789431572.134003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431572.262160:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913530240:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431572.262251:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913530496:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431572.262284:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789431572.262291:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913530368:192.168.203.128@tcp:400:kworker.0 00010000:00080000:0.0:1789431572.266786:0:2390:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d0bdea00 x1876350913530496/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431588 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431572.266805:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913530496:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431572.267160:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913530368:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431572.272300:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913530240:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431573.159123:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1789431573.496872:0:332287:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/symlink/12.t against lustre filesystem 00000080:00200000:1.0:1789431573.685146:0:332299:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431573.685176:0:332299:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431573.685193:0:332299:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431573.685207:0:332299:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431573.685215:0:332299:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431573.685221:0:332299:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431573.685225:0:332299:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:1.0:1789431573.685232:0:332299:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1bfbb58 00010000:00010000:1.0:1789431573.685236:0:332299:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431573.685244:0:332299:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431573.685260:0:332299:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.685267:0:332299:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=13 00000080:00200008:1.0:1789431573.685276:0:332299:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431573.685281:0:332299:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431573.685292:0:332299:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431573.685301:0:332299:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1789431573.685309:0:332299:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00000080:00200000:1.0:1789431573.685315:0:332299:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431573.685319:0:332299:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431573.685321:0:332299:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431573.685325:0:332299:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1bfbcb8 00010000:00010000:1.0:1789431573.685327:0:332299:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 58 lru_type: 1 00010000:00010000:1.0:1789431573.685333:0:332299:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000080:00200008:1.0:1789431573.685793:0:332299:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431573.685798:0:332299:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431573.685808:0:332299:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431573.685816:0:332299:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431573.685825:0:332299:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431573.685849:0:332299:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431573.685853:0:332299:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431573.685856:0:332299:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:1.0:1789431573.685860:0:332299:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1bfbcb8 00010000:00010000:1.0:1789431573.685864:0:332299:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431573.685871:0:332299:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431573.686174:0:332299:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431573.686183:0:332299:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431573.686190:0:332299:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431573.686198:0:332299:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431573.686210:0:332299:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431573.686214:0:332299:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431573.686216:0:332299:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea4c 00000080:00010000:1.0:1789431573.686220:0:332299:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea4c from it ffffbc42c1bfbbb0 00010000:00010000:1.0:1789431573.686224:0:332299:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431573.686231:0:332299:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:1.0:1789431573.686240:0:332299:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:1.0:1789431573.686245:0:332299:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:1.0:1789431573.686258:0:332299:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffde3267320 00000080:00200000:1.0:1789431573.686270:0:332299:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:1.0:1789431573.686289:0:332299:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431573.686329:0:332299:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431573.686336:0:332299:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1789431573.686339:0:332299:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119c00/0x1ffb2ee1bd27ea4c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad337 expref: -99 pid: 330668 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:1.0:1789431573.686359:0:332299:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431573.686369:0:332299:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bddf80 x1876350913530752/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431573.686390:0:332299:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332299:1876350913530752:192.168.203.128@tcp:36:lfs.0 00000100:00100000:1.0:1789431573.687715:0:332299:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431573.694488:0:332299:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332299:1876350913530752:192.168.203.128@tcp:36:lfs.0 00000080:00200008:1.0:1789431573.694619:0:332299:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200008:0.0:1789431573.805551:0:331831:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431573.805562:0:331831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431573.805584:0:331831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431573.805639:0:331831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 331831 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.805649:0:331831:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 331831 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.805658:0:331831:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 331831 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431573.805680:0:331831:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331831:1876350913530880:192.168.203.128@tcp:101:bash.0 00000100:00100000:0.0:1789431573.805765:0:331831:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431573.809989:0:331831:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:331831:1876350913530880:192.168.203.128@tcp:101:bash.0 00010000:00010000:0.0:1789431573.810017:0:331831:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810031:0:331831:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810044:0:331831:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810120:0:331831:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810145:0:331831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810153:0:331831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810161:0:331831:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431573.810174:0:331831:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811c8cd4380 x1876350913530880/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431573.810188:0:331831:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810200:0:331831:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1789431573.810207:0:331831:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1789431573.810235:0:331831:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:0.0:1789431573.810242:0:331831:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:0.0:1789431573.810247:0:331831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431573.810250:0:331831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431573.810254:0:331831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431573.810263:0:331831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431573.810315:0:331831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431573.810331:0:331831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431573.810339:0:331831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431573.810350:0:331831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431573.810357:0:331831:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431573.810361:0:331831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431573.810364:0:331831:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:0.0:1789431573.810367:0:331831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0da3b58 00010000:00010000:0.0:1789431573.810371:0:331831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431573.810379:0:331831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431573.810387:0:331831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431573.810394:0:331831:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:0.0:1789431573.810403:0:331831:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431573.810407:0:331831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431573.810419:0:331831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431573.810426:0:331831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:0.0:1789431573.810434:0:331831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00000080:00200000:0.0:1789431573.810441:0:331831:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431573.810445:0:331831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431573.810448:0:331831:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431573.810452:0:331831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0da3cb8 00010000:00010000:0.0:1789431573.810455:0:331831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 59 lru_type: 1 00010000:00010000:0.0:1789431573.810462:0:331831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000080:00200000:0.0:1789431573.810481:0:331831:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431573.810488:0:331831:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431573.810494:0:331831:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431573.810501:0:331831:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1789431573.810508:0:331831:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431573.810511:0:331831:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431573.810513:0:331831:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:0.0:1789431573.810517:0:331831:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0da3c18 00010000:00010000:0.0:1789431573.810520:0:331831:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431573.810527:0:331831:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1789431573.810535:0:331831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431573.810538:0:331831:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:0.0:1789431573.810543:0:331831:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:1.0:1789431573.815163:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815180:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.815191:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815195:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815202:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815204:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815211:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815213:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815220:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815222:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815229:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815232:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815238:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815240:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815246:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815248:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815254:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815258:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:1.0:1789431573.815454:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815459:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.815467:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815470:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815478:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815480:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815488:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815491:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815499:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815502:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815510:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815512:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815520:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815523:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815530:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815532:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815541:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815545:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431573.815696:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815701:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.815711:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815714:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815722:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815725:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815733:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815737:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815744:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815748:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.815756:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815759:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815766:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815769:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815775:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815778:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.815786:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815789:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:1.0:1789431573.815985:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.815991:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.816002:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816006:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816014:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816017:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816024:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816027:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816157:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816162:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816172:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816175:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816184:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816187:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816194:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816197:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816205:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816208:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431573.816326:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816332:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.816342:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816346:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816355:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816358:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816367:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816370:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816378:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816381:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816390:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816393:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816400:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816403:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816413:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816416:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816424:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816427:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431573.816546:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816551:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.816561:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816565:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816576:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816579:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816589:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816592:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816600:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816603:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816612:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816616:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816626:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816630:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816638:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816641:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816651:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816654:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:1.0:1789431573.816776:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816782:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.816792:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816794:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816964:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816968:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816974:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816975:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816980:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816982:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.816987:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816989:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.816993:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.816994:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.817000:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817001:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.817006:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817008:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:1.0:1789431573.817328:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817333:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431573.817342:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817346:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.817354:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817357:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.817366:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817369:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.817380:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817383:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431573.817394:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817397:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.817406:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817409:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.817418:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817421:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431573.817430:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431573.817433:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431573.956516:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956532:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431573.956544:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956548:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431573.956557:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956560:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431573.956568:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956571:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431573.956578:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956581:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431573.956587:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956591:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431573.956598:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956601:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431573.956609:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956612:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431573.956619:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.956622:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=librt.so.1, flags=257 00000080:00200000:2.0:1789431573.977458:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431573.977475:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.proverc, flags=1 00000080:00200000:3.0:1789431573.979629:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431573.979643:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.prove, flags=1 00000080:00200000:2.0:1789431574.013597:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.013610:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.013616:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=testrules.yml, flags=1 00000080:00200000:2.0:1789431574.013628:0:332303:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.013631:0:332303:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t, flags=17 00000080:00200000:1.0:1789431574.150379:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150396:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.150411:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150423:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150435:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150438:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150446:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150449:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150458:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150461:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150470:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150473:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.150480:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150483:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.150496:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150499:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.150687:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150694:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:1.0:1789431574.150939:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150944:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.150955:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150959:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150968:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.150972:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.150998:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151002:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.151013:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151016:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.151025:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151029:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.151039:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151042:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.151064:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151067:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.151075:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.151080:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431574.152055:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152064:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.152077:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152091:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152325:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152331:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152341:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152344:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152351:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152355:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152363:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152366:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152373:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152376:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152385:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152388:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152395:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152399:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:1.0:1789431574.152565:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152582:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.152595:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152599:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152606:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152609:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152616:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152619:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152627:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152631:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.152639:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152642:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152652:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152655:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152662:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152665:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.152672:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152676:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431574.152810:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152815:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.152823:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.152826:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153088:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153094:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153345:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153351:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153361:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153364:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153372:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153376:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153384:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153387:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153396:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153399:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153407:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153411:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431574.153551:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153557:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.153566:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153634:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153651:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153654:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153665:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153669:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153677:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153679:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.153687:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153689:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153698:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153701:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153711:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153714:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.153723:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153726:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:1.0:1789431574.153994:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.153999:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.154007:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154010:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154018:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154021:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154030:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154032:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154039:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154042:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154049:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154052:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154062:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154065:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154076:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154079:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154334:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154340:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:1.0:1789431574.154483:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154488:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.154496:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154499:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154505:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154508:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154515:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154518:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154525:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154528:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.154538:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154541:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154549:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154552:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154560:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154563:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.154654:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.154660:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.163131:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163147:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.163161:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163165:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163175:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163177:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163185:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163187:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163193:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163196:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163202:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163205:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163211:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163214:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163219:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163222:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163229:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163232:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:1.0:1789431574.163391:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163396:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.163403:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163406:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163413:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163415:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163423:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163425:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163432:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163435:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163443:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163446:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163453:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163455:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163462:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163464:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163472:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163476:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431574.163581:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163586:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.163596:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163599:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163607:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163610:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163618:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163622:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163629:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163632:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.163641:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163645:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163653:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163656:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163665:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163668:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.163678:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.163681:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.165742:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431574.165774:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.165791:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431574.165807:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1789431574.165815:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.165822:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431574.165825:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431574.165858:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0dd3b58 00010000:00010000:1.0:1789431574.165862:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1789431574.165872:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431574.165887:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.165895:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431574.165904:0:332304:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.165910:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.165922:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.165944:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1789431574.165953:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00000080:00200000:1.0:1789431574.165960:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.165964:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.165966:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.165970:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0dd3cb8 00010000:00010000:1.0:1789431574.165973:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 60 lru_type: 1 00010000:00010000:1.0:1789431574.165979:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000080:00200000:1.0:1789431574.165995:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431574.166000:0:332304:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.166004:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.166012:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.166018:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1789431574.166026:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00000080:00200000:1.0:1789431574.166038:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.166043:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.166045:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.166049:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0dd3cb8 00010000:00010000:1.0:1789431574.166052:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 61 lru_type: 1 00010000:00010000:1.0:1789431574.166059:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000080:00200008:1.0:1789431574.166080:0:332304:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.166084:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431574.166089:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.166095:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431574.166102:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1789431574.166108:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.166111:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431574.166113:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431574.166117:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0dd3cb8 00010000:00010000:1.0:1789431574.166120:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431574.166130:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431574.166144:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431574.166149:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.166155:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431574.166162:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1789431574.166169:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.166173:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431574.166175:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431574.166178:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0dd3b58 00010000:00010000:1.0:1789431574.166181:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1789431574.166187:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431574.166195:0:332304:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.166198:0:332304:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:1.0:1789431574.166203:0:332304:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.166206:0:332304:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.166212:0:332304:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.166218:0:332304:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1789431574.166225:0:332304:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00000080:00200000:1.0:1789431574.166231:0:332304:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.166235:0:332304:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.166237:0:332304:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.166240:0:332304:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0dd3cb8 00010000:00010000:1.0:1789431574.166243:0:332304:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 62 lru_type: 1 00010000:00010000:1.0:1789431574.166249:0:332304:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000080:00200000:0.0:1789431574.167892:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.167907:0:332305:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.167913:0:332305:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431574.167945:0:332305:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.167963:0:332305:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:0.0:1789431574.167976:0:332305:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00000080:00200000:0.0:1789431574.167984:0:332305:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.167990:0:332305:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.167993:0:332305:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431574.168001:0:332305:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d7bcb8 00010000:00010000:0.0:1789431574.168004:0:332305:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 63 lru_type: 1 00010000:00010000:0.0:1789431574.168013:0:332305:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000080:00200000:0.0:1789431574.168811:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168819:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.168826:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168828:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.168862:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168866:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.168875:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168878:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.168886:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168889:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.168895:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168898:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.168903:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168905:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.168909:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168911:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.168915:0:332305:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.168917:0:332305:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.176175:0:332308:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431574.176192:0:332308:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.176198:0:332308:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431574.176224:0:332308:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431574.176241:0:332308:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:2.0:1789431574.176256:0:332308:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00000080:00200000:2.0:1789431574.176268:0:332308:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431574.176276:0:332308:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431574.176279:0:332308:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431574.176287:0:332308:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d7bcb8 00010000:00010000:2.0:1789431574.176291:0:332308:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 64 lru_type: 1 00010000:00010000:2.0:1789431574.176300:0:332308:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000080:00200000:0.0:1789431574.179804:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.179821:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.179993:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180000:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.180011:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180014:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.180021:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180024:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.180031:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180034:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.180041:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180044:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.180052:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180054:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.180062:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180066:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.180074:0:332309:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.180077:0:332309:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00010000:00010000:1.0:1789431574.181983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1789431574.183790:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.183809:0:332311:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.183816:0:332311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431574.183980:0:332311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.183999:0:332311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:0.0:1789431574.184015:0:332311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00000080:00200000:0.0:1789431574.184024:0:332311:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.184032:0:332311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.184036:0:332311:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431574.184044:0:332311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dcbcb8 00010000:00010000:0.0:1789431574.184048:0:332311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 65 lru_type: 1 00010000:00010000:0.0:1789431574.184057:0:332311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000080:00200000:2.0:1789431574.184850:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431574.184862:0:332313:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.184867:0:332313:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431574.184880:0:332313:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431574.184891:0:332313:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:2.0:1789431574.184914:0:332313:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00000080:00200000:2.0:1789431574.184921:0:332313:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431574.184938:0:332313:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431574.184941:0:332313:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431574.184946:0:332313:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1677cb8 00010000:00010000:2.0:1789431574.184949:0:332313:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 66 lru_type: 1 00010000:00010000:2.0:1789431574.184962:0:332313:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:0.0:1789431574.185317:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185324:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.185333:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185337:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.185343:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185346:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.185353:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185356:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.185363:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185367:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.185374:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185378:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.185386:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185388:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.185395:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185398:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.185405:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.185409:0:332311:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.185975:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.185981:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.185991:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.185994:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186005:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186008:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186016:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186019:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186029:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186032:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186041:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186044:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186051:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186054:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186062:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186065:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186074:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186077:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:2.0:1789431574.186204:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186209:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.186218:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186221:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186230:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186233:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.186237:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186241:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.186243:0:332311:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.186244:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.186248:0:332311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000080:00200000:2.0:1789431574.186252:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186255:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000002:00010000:0.0:1789431574.186261:0:332311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:2.0:1789431574.186263:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186266:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00010000:00010000:0.0:1789431574.186274:0:332311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:2.0:1789431574.186275:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186278:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00010000:00010000:0.0:1789431574.186287:0:332311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00000080:00200000:2.0:1789431574.186288:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186291:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.186295:0:332311:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00200000:2.0:1789431574.186299:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00010000:0.0:1789431574.186300:0:332311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.186303:0:332311:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00200000:2.0:1789431574.186303:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00010000:0.0:1789431574.186307:0:332311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dcbcb8 00010000:00010000:0.0:1789431574.186312:0:332311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 67 lru_type: 1 00010000:00010000:0.0:1789431574.186320:0:332311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:0.0:1789431574.186340:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.186344:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:0.0:1789431574.186349:0:332311:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:0.0:1789431574.186364:0:332311:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:0.0:1789431574.186376:0:332311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1789431574.186447:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186452:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186459:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200008:3.0:1789431574.186460:0:332312:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.186468:0:332312:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000080:00200000:2.0:1789431574.186469:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186476:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000002:00010000:3.0:1789431574.186486:0:332312:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:2.0:1789431574.186489:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186496:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00010000:00010000:3.0:1789431574.186501:0:332312:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:2.0:1789431574.186505:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186510:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186517:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186521:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186532:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00010000:00010000:3.0:1789431574.186533:0:332312:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:2.0:1789431574.186538:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.186546:0:332312:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00200000:2.0:1789431574.186550:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00010000:3.0:1789431574.186552:0:332312:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00200000:2.0:1789431574.186555:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00010000:3.0:1789431574.186557:0:332312:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.186564:0:332312:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1453cb8 00000080:00200000:2.0:1789431574.186566:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00010000:00010000:3.0:1789431574.186569:0:332312:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 68 lru_type: 1 00000080:00200000:2.0:1789431574.186571:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186582:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00010000:00010000:3.0:1789431574.186584:0:332312:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000080:00200000:2.0:1789431574.186588:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:2.0:1789431574.186737:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186742:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.186754:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186758:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186767:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186770:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186778:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186781:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186790:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186793:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.186806:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186809:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186820:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186823:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186855:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186859:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.186872:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.186876:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431574.187027:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187032:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.187045:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187049:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187065:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187068:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187076:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187078:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187086:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187089:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187098:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187101:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187110:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187112:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187124:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187128:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187137:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187141:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431574.187292:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187297:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.187309:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187312:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187322:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187325:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187332:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187335:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187349:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187351:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187365:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187368:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187378:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187381:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187389:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187392:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187402:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187406:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431574.187565:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187571:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.187581:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187585:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187597:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187600:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187610:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187613:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187622:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187625:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.187634:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187637:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187644:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187647:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187655:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187658:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.187667:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.187670:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.187923:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.189123:0:332311:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.189130:0:332311:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:0.0:1789431574.189149:0:332311:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.189166:0:332311:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431574.189180:0:332311:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:0.0:1789431574.189187:0:332311:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.189194:0:332311:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:0.0:1789431574.189197:0:332311:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:0.0:1789431574.189204:0:332311:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c1dcbcb8 00010000:00010000:0.0:1789431574.189208:0:332311:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:0.0:1789431574.189222:0:332311:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:0.0:1789431574.189247:0:332311:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.189253:0:332311:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:0.0:1789431574.189257:0:332311:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3935, max_age 3947 00800000:00000004:0.0:1789431574.189267:0:332311:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3935, max_age 3947 00000100:00100000:0.0:1789431574.189304:0:332311:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332311:1876350913531008:192.168.203.128@tcp:41:df.0 00000100:00100000:0.0:1789431574.189391:0:332311:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9480 going to sleep for 11 seconds 00000080:00200000:0.0:1789431574.191171:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191184:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.191195:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191199:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.191205:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191208:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.191214:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191216:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.191222:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191224:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.191230:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191234:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.191241:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191243:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.191249:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191252:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.191259:0:332312:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.191262:0:332312:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.191670:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.191695:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191700:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.191711:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191714:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.191723:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191726:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.191736:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191739:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.191745:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191748:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.191757:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191760:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.191767:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191770:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.191778:0:332313:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.191782:0:332313:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000100:00100000:0.0:1789431574.194682:0:332311:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332311:1876350913531008:192.168.203.128@tcp:41:df.0 00800000:00000004:0.0:1789431574.194707:0:332311:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:0.0:1789431574.194713:0:332311:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:0.0:1789431574.194718:0:332311:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000080:00200000:2.0:1789431574.204331:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431574.204351:0:332316:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.204358:0:332316:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431574.204382:0:332316:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431574.204407:0:332316:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:2.0:1789431574.204421:0:332316:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00000080:00200000:2.0:1789431574.204430:0:332316:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431574.204438:0:332316:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431574.204442:0:332316:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431574.204449:0:332316:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dcbcb8 00010000:00010000:2.0:1789431574.204453:0:332316:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 69 lru_type: 1 00010000:00010000:2.0:1789431574.204462:0:332316:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000080:00200000:1.0:1789431574.204786:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431574.204795:0:332317:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.204799:0:332317:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.204822:0:332317:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.204856:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1789431574.204874:0:332317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00000080:00200000:1.0:1789431574.204883:0:332317:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.204889:0:332317:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.204892:0:332317:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.204897:0:332317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174bcb8 00010000:00010000:1.0:1789431574.204900:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 70 lru_type: 1 00010000:00010000:1.0:1789431574.204908:0:332317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:1.0:1789431574.206146:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206157:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.206170:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206174:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.206184:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206187:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.206195:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206198:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.206201:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206208:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206212:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200008:3.0:1789431574.206212:0:332318:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.206222:0:332318:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000080:00200000:1.0:1789431574.206223:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206229:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.206237:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206239:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000002:00010000:3.0:1789431574.206244:0:332318:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:1.0:1789431574.206245:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.206249:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.206257:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00010000:00010000:3.0:1789431574.206263:0:332318:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000080:00200000:1.0:1789431574.206264:0:332317:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d0acd080, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431574.206273:0:332317:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libssl.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00010000:00010000:3.0:1789431574.206283:0:332318:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00000002:00010000:1.0:1789431574.206290:0:332317:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libssl.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00000080:00200000:3.0:1789431574.206293:0:332318:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431574.206301:0:332318:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431574.206304:0:332318:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.206309:0:332318:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1753cb8 00010000:00010000:3.0:1789431574.206313:0:332318:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 71 lru_type: 1 00010000:00010000:3.0:1789431574.206322:0:332318:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1789431574.206342:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.206352:0:332317:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431574.206363:0:332317:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431574.206376:0:332317:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bddf80 x1876350913531264/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431574.206386:0:332317:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431574.206406:0:332317:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531264:192.168.203.128@tcp:101:openssl.0 00000100:00100000:1.0:1789431574.206566:0:332317:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64600 going to sleep for 11 seconds 00000080:00200000:3.0:1789431574.207670:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207681:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.207691:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207695:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.207703:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207706:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.207712:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207715:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.207723:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207725:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.207735:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207738:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.207746:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207750:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.207759:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207762:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.207770:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207774:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:3.0:1789431574.207960:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207966:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.207976:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207979:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.207989:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.207992:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208000:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208003:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208010:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208013:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208019:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208022:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208029:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208032:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208038:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208041:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208048:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208051:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:3.0:1789431574.208172:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208177:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.208185:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208188:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208195:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208198:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208204:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208207:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208215:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208217:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208224:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208226:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208234:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208237:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208245:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208248:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208257:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208260:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:3.0:1789431574.208380:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208385:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.208394:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208397:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208404:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208407:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208414:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208416:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208424:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208426:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208435:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208438:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208445:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208447:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208454:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208456:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208463:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208466:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:3.0:1789431574.208584:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208589:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.208598:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208601:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208609:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208612:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208619:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208622:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208630:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208633:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208640:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208643:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208650:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208652:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208660:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208662:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208671:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208674:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431574.208791:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208797:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.208806:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208809:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208817:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208819:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208827:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208829:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208861:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208865:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.208874:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208877:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208885:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208888:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208895:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208897:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.208906:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.208909:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431574.209355:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209362:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.209373:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209376:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.209385:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209389:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.209398:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209402:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.209410:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209413:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.209421:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209424:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.209432:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209436:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.209445:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209448:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.209457:0:332316:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.209461:0:332316:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.211231:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211241:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.211251:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211255:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211262:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211265:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211272:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211276:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211283:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211286:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211293:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211297:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211304:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211307:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211315:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211317:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211325:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211328:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.211486:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211491:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.211501:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211505:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211512:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211515:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211523:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211526:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211534:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211537:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.211545:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211548:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211559:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211562:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211571:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211574:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.211582:0:332318:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.211585:0:332318:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000100:00100000:1.0:1789431574.212132:0:332317:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531264:192.168.203.128@tcp:101:openssl.0 00010000:00010000:1.0:1789431574.212152:0:332317:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212164:0:332317:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212171:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212178:0:332317:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212188:0:332317:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212199:0:332317:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fc00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431574.212204:0:332317:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212211:0:332317:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212218:0:332317:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212225:0:332317:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431574.212231:0:332317:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212237:0:332317:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fc00/0x1ffb2ee1bd27ea5a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431574.212254:0:332317:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bddf80 x1876350913531264/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431590 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431574.212267:0:332317:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431574.212291:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1789431574.212303:0:332317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00000080:00010000:1.0:1789431574.212312:0:332317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174b9a0 00010000:00010000:1.0:1789431574.212316:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 72 lru_type: 1 00010000:00010000:1.0:1789431574.212329:0:332317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00200000:1.0:1789431574.212548:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212555:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.212563:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212567:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.212574:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212577:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.212585:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212588:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.212596:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212599:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.212606:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212609:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.212620:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212623:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.212630:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212632:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.212640:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.212649:0:332317:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811ca227380, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431574.212655:0:332317:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libcrypto.so.1.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:1.0:1789431574.212673:0:332317:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libcrypto.so.1.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431574.212716:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.212724:0:332317:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431574.212734:0:332317:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431574.212740:0:332317:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bddf80 x1876350913531520/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431574.212749:0:332317:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431574.212768:0:332317:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531520:192.168.203.128@tcp:101:openssl.0 00000100:00100000:1.0:1789431574.215237:0:332317:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0a64600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431574.219890:0:332317:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531520:192.168.203.128@tcp:101:openssl.0 00010000:00010000:1.0:1789431574.220122:0:332317:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220136:0:332317:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220144:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220152:0:332317:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220161:0:332317:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220167:0:332317:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ee00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431574.220172:0:332317:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220188:0:332317:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220196:0:332317:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220202:0:332317:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431574.220208:0:332317:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431574.220215:0:332317:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ee00/0x1ffb2ee1bd27ea61 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431574.220233:0:332317:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bddf80 x1876350913531520/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 664/608 e 0 to 0 dl 1789431590 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431574.220246:0:332317:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431574.220464:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:0.0:1789431574.220478:0:332317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00000080:00010000:0.0:1789431574.220488:0:332317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174b9a0 00010000:00010000:0.0:1789431574.220492:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 73 lru_type: 1 00010000:00010000:0.0:1789431574.220500:0:332317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00200000:0.0:1789431574.220820:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.220829:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.220880:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.220884:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.220891:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.220894:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.221572:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221578:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.221586:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221589:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.221596:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221600:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.221607:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221610:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.221617:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221620:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.221627:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.221637:0:332317:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c5a92a80, open_flags 8000, mode 0 opened 0 00000080:00200000:0.0:1789431574.221643:0:332317:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libz.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:0.0:1789431574.221660:0:332317:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libz.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:0.0:1789431574.221704:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.221712:0:332317:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431574.221722:0:332317:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431574.221731:0:332317:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4380 x1876350913531776/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431574.221740:0:332317:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431574.221760:0:332317:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531776:192.168.203.128@tcp:101:openssl.0 00000100:00100000:0.0:1789431574.222600:0:332317:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431574.226515:0:332317:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job openssl:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:332317:1876350913531776:192.168.203.128@tcp:101:openssl.0 00010000:00010000:0.0:1789431574.226533:0:332317:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226542:0:332317:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226550:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226555:0:332317:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226560:0:332317:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226564:0:332317:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9812021c2400 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:0.0:1789431574.226567:0:332317:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226571:0:332317:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226576:0:332317:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226580:0:332317:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1789431574.226584:0:332317:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431574.226588:0:332317:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2400/0x1ffb2ee1bd27ea68 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 332317 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431574.226600:0:332317:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811c8cd4380 x1876350913531776/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431590 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'openssl.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1789431574.226613:0:332317:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:0.0:1789431574.226639:0:332317:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:0.0:1789431574.226653:0:332317:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00000080:00010000:0.0:1789431574.226660:0:332317:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174b9a0 00010000:00010000:0.0:1789431574.226663:0:332317:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 74 lru_type: 1 00010000:00010000:0.0:1789431574.226669:0:332317:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:0.0:1789431574.227037:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227046:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.227070:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227075:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227084:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227088:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227096:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227099:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227112:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227115:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227124:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227128:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227135:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227138:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227146:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227149:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227158:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227162:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431574.227287:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227293:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.227303:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227306:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227315:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227318:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227326:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227329:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227336:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227339:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227347:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227350:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227358:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227361:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227369:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227373:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227380:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227383:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431574.227517:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227523:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.227534:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227537:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227546:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227549:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227557:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227560:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227567:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227570:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.227577:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227579:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227586:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227589:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227599:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227602:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.227610:0:332317:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.227614:0:332317:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.238619:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431574.238637:0:332321:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.238645:0:332321:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.238669:0:332321:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.238688:0:332321:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1789431574.238703:0:332321:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00000080:00200000:1.0:1789431574.238711:0:332321:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.238718:0:332321:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.238722:0:332321:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.238729:0:332321:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1593cb8 00010000:00010000:1.0:1789431574.238733:0:332321:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 75 lru_type: 1 00010000:00010000:1.0:1789431574.238742:0:332321:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000080:00200000:2.0:1789431574.239036:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239045:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.239058:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239062:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.239071:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239075:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.239083:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239087:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.239096:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239100:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.239113:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239118:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.239132:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239135:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.239151:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239155:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.239170:0:332320:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.239175:0:332320:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.240448:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240456:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.240468:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240471:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.240479:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240481:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.240489:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240492:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.240505:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240508:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.240517:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240520:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.240528:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240531:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.240539:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240542:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.240549:0:332321:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.240552:0:332321:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.243993:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431574.244009:0:332323:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.244017:0:332323:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431574.244042:0:332323:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431574.244060:0:332323:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:3.0:1789431574.244075:0:332323:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00000080:00200000:3.0:1789431574.244084:0:332323:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431574.244091:0:332323:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431574.244095:0:332323:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.244102:0:332323:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dcbcb8 00010000:00010000:3.0:1789431574.244107:0:332323:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 76 lru_type: 1 00010000:00010000:3.0:1789431574.244117:0:332323:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000080:00200000:3.0:1789431574.245438:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245450:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.245462:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245467:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245475:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245478:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245486:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245489:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245497:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245500:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245508:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245511:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245518:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245521:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245528:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245531:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245539:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245542:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431574.245710:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245715:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.245725:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245729:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245737:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245740:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245748:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245751:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245759:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245761:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.245769:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245772:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245780:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245784:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245793:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245796:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.245804:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245807:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.245983:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245988:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.245996:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.245999:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.246005:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246008:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.246014:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246017:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.246023:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246026:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.246034:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246037:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.246044:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246047:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.246053:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246056:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.246064:0:332323:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.246066:0:332323:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431574.252079:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.252095:0:332326:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.252102:0:332326:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431574.252126:0:332326:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.252144:0:332326:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:0.0:1789431574.252159:0:332326:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00000080:00200000:0.0:1789431574.252166:0:332326:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.252172:0:332326:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.252176:0:332326:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431574.252182:0:332326:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1d9bcb8 00010000:00010000:0.0:1789431574.252186:0:332326:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 77 lru_type: 1 00010000:00010000:0.0:1789431574.252194:0:332326:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000080:00200000:2.0:1789431574.253199:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253211:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.253228:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253233:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.253245:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253248:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.253257:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253260:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.253271:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253279:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.253289:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253292:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.253301:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253304:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.253314:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253318:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.253328:0:332325:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.253333:0:332325:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431574.253571:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253577:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.253587:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253591:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.253600:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253603:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.253612:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253615:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.253623:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253626:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.253633:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253636:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.253645:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253648:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.253655:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253659:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.253666:0:332326:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.253669:0:332326:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.257586:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431574.257603:0:332328:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.257610:0:332328:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431574.257633:0:332328:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431574.257649:0:332328:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:3.0:1789431574.257663:0:332328:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00000080:00200000:3.0:1789431574.257677:0:332328:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431574.257683:0:332328:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431574.257687:0:332328:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.257694:0:332328:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1bebcb8 00010000:00010000:3.0:1789431574.257697:0:332328:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 78 lru_type: 1 00010000:00010000:3.0:1789431574.257706:0:332328:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000080:00200000:1.0:1789431574.259995:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260009:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.260023:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260028:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260038:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260042:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260049:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260052:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260059:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260062:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260070:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260073:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260080:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260083:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260092:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260095:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260103:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260107:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:1.0:1789431574.260277:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260282:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.260292:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260295:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260305:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260308:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260316:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260319:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260327:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260330:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260338:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260341:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260348:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260351:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260360:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260363:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260370:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260374:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.260499:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260504:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.260513:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260516:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260523:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260526:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260532:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260534:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260541:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260544:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.260550:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260553:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260561:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260563:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260569:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260572:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.260578:0:332328:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.260581:0:332328:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431574.266726:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431574.266744:0:332331:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.266751:0:332331:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431574.266779:0:332331:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431574.266797:0:332331:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:2.0:1789431574.266812:0:332331:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00000080:00200000:2.0:1789431574.266820:0:332331:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431574.266827:0:332331:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431574.267077:0:332331:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431574.267086:0:332331:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dc3cb8 00010000:00010000:2.0:1789431574.267091:0:332331:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 79 lru_type: 1 00010000:00010000:2.0:1789431574.267101:0:332331:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000080:00200000:0.0:1789431574.267501:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267510:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.267523:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267528:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.267540:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267543:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.267553:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267556:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.267566:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267569:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.267579:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267583:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.267595:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267598:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.267611:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267615:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.267627:0:332330:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.267631:0:332330:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.268619:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268628:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.268642:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268647:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.268658:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268662:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.268677:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268679:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.268688:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268691:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.268702:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268704:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.268714:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268717:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.268726:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268728:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.268733:0:332331:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.268735:0:332331:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.273717:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431574.273734:0:332333:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.273741:0:332333:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.273766:0:332333:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.273784:0:332333:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1789431574.273812:0:332333:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00000080:00200000:1.0:1789431574.273821:0:332333:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.273829:0:332333:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.273852:0:332333:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.273860:0:332333:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174bcb8 00010000:00010000:1.0:1789431574.273863:0:332333:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 80 lru_type: 1 00010000:00010000:1.0:1789431574.273874:0:332333:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000080:00200000:1.0:1789431574.276322:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276335:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.276346:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276350:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276358:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276361:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276370:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276373:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276381:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276384:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276391:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276395:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276402:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276405:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276413:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276416:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276423:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276427:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:1.0:1789431574.276549:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276553:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.276559:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276561:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276566:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276567:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276572:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276573:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276577:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276579:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276583:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276585:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276589:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276590:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276595:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276596:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276600:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276602:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.276710:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276716:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.276725:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276728:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276736:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276739:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276747:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276750:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276757:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276759:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.276767:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276770:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276785:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276788:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276799:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276802:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.276810:0:332333:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.276813:0:332333:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431574.281826:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.282018:0:332336:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.282025:0:332336:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431574.282061:0:332336:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.282079:0:332336:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:0.0:1789431574.282094:0:332336:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00000080:00200000:0.0:1789431574.282102:0:332336:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.282109:0:332336:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.282112:0:332336:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431574.282119:0:332336:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1453cb8 00010000:00010000:0.0:1789431574.282123:0:332336:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 81 lru_type: 1 00010000:00010000:0.0:1789431574.282131:0:332336:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000080:00200000:2.0:1789431574.282510:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282518:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.282528:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282531:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.282538:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282541:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.282548:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282551:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.282557:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282560:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.282567:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282571:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.282578:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282581:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.282588:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282590:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.282598:0:332335:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.282601:0:332335:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431574.283202:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283210:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.283219:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283223:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.283232:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283235:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.283243:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283246:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.283254:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283257:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.283263:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283266:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.283272:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283275:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.283283:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283286:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.283293:0:332336:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.283296:0:332336:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.286386:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431574.286403:0:332338:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.286410:0:332338:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431574.286441:0:332338:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431574.286458:0:332338:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:3.0:1789431574.286473:0:332338:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00000080:00200000:3.0:1789431574.286482:0:332338:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431574.286488:0:332338:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431574.286491:0:332338:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.286498:0:332338:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c178bcb8 00010000:00010000:3.0:1789431574.286502:0:332338:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 82 lru_type: 1 00010000:00010000:3.0:1789431574.286517:0:332338:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000080:00200000:3.0:1789431574.287722:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287731:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.287737:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287740:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.287745:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287746:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.287751:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287753:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.287758:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287760:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.287768:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287771:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.287779:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287782:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.287788:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287791:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.287798:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.287802:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431574.288036:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288040:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.288045:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288056:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288063:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288065:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288070:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288072:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288077:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288079:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288084:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288086:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288090:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288092:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288097:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288099:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288106:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288110:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431574.288221:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288225:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.288230:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288232:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288237:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288238:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288243:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288245:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288249:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288251:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.288255:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288257:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288261:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288263:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288267:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288268:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.288272:0:332338:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.288274:0:332338:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431574.292325:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431574.292343:0:332341:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431574.292350:0:332341:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431574.292375:0:332341:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431574.292391:0:332341:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1789431574.292407:0:332341:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00000080:00200000:1.0:1789431574.292415:0:332341:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431574.292423:0:332341:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431574.292427:0:332341:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431574.292433:0:332341:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1dc3cb8 00010000:00010000:1.0:1789431574.292437:0:332341:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 83 lru_type: 1 00010000:00010000:1.0:1789431574.292444:0:332341:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000080:00200000:3.0:1789431574.293262:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293272:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.293283:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293287:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.293295:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293298:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.293305:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293308:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.293315:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293319:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.293326:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293330:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.293338:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293341:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.293349:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293352:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.293360:0:332340:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.293364:0:332340:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431574.293507:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293512:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431574.293520:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293524:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.293531:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293534:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.293542:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293546:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.293553:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293556:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431574.293565:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293568:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.293577:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293579:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.293586:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293588:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431574.293594:0:332341:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431574.293597:0:332341:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431574.299455:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431574.299474:0:332343:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431574.299480:0:332343:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431574.299506:0:332343:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431574.299523:0:332343:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:0.0:1789431574.299537:0:332343:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00000080:00200000:0.0:1789431574.299544:0:332343:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431574.299620:0:332343:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431574.299624:0:332343:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431574.299630:0:332343:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1677cb8 00010000:00010000:0.0:1789431574.299635:0:332343:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 84 lru_type: 1 00010000:00010000:0.0:1789431574.299644:0:332343:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:0.0:1789431574.301329:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301340:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.301347:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301349:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301354:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301356:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301360:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301362:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301367:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301369:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301373:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301375:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301380:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301381:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301386:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301387:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301391:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301393:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:0.0:1789431574.301511:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301515:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.301520:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301521:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301526:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301528:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301532:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301534:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301539:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301540:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301545:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301547:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301551:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301553:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301557:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301559:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301563:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301565:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431574.301653:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301657:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.301664:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301667:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301674:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301677:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301683:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301686:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301694:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301696:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.301704:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301707:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301716:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301719:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301727:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301730:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.301735:0:332343:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.301737:0:332343:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431574.307506:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431574.307522:0:332346:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431574.307529:0:332346:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431574.307553:0:332346:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431574.307569:0:332346:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:3.0:1789431574.307584:0:332346:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00000080:00200000:3.0:1789431574.307592:0:332346:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431574.307599:0:332346:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431574.307603:0:332346:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431574.307609:0:332346:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c1bcb8 00010000:00010000:3.0:1789431574.307613:0:332346:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 85 lru_type: 1 00010000:00010000:3.0:1789431574.307622:0:332346:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:3.0:1789431574.308681:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308690:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431574.308699:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308702:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.308709:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308711:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.308718:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308721:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.308727:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308730:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431574.308737:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308740:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.308746:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308749:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.308755:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308757:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431574.308764:0:332346:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431574.308767:0:332346:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431574.310215:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310227:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431574.310237:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310240:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.310246:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310249:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.310257:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310260:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.310267:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310270:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431574.310277:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310280:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.310287:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310290:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.310298:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310301:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431574.310309:0:332345:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431574.310313:0:332345:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.314023:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431574.314035:0:332348:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431574.314040:0:332348:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431574.314062:0:332348:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431574.314086:0:332348:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1789431574.314649:0:332348:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00000080:00200000:2.0:1789431574.314657:0:332348:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431574.314664:0:332348:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431574.314666:0:332348:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431574.314672:0:332348:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1c43cb8 00010000:00010000:2.0:1789431574.314675:0:332348:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 86 lru_type: 1 00010000:00010000:2.0:1789431574.314681:0:332348:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:2.0:1789431574.316386:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316399:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.316412:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316416:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316424:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316427:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316435:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316438:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316446:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316449:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316456:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316459:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316467:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316471:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316477:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316480:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316487:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316490:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431574.316647:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316651:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.316658:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316660:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316665:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316667:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316671:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316673:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316677:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316679:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316683:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316685:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316689:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316690:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316694:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316697:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316701:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316703:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431574.316813:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316818:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431574.316828:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316862:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316876:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316879:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316888:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316890:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316897:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316900:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431574.316909:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316912:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316920:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316923:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316946:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316950:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431574.316960:0:332348:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431574.316964:0:332348:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000001:02000400:3.0:1789431575.203829:0:332480:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: ext4 report 00010000:00010000:2.0:1789431575.205968:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:0.0:1789431575.989446:0:332619:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: lustre report 00010000:00010000:3.0:1789431576.230000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431577.254964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431577.383007:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913531904:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431577.383039:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913532032:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431577.383109:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913532160:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431577.384528:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913532032:192.168.203.128@tcp:400:kworker.0 00010000:00080000:2.0:1789431577.384559:0:2391:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d2103800 x1876350913532160/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431593 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431577.384570:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913532160:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431577.384580:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913531904:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431578.278946:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1789431578.545381:0:333196:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == run_pjdfstest test truncate_00: truncate descrease/increase file size ========================================================== 20:19:37 (1789431577) 00000001:02000400:1.0:1789431579.164304:0:333334:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/truncate/00.t against ext4 filesystem 00010000:00010000:2.0:1789431579.302001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431580.326046:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431581.350167:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1789431582.354937:0:333599:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: Run /usr/share/pjdfstest/truncate/00.t against lustre filesystem 00010000:00010000:2.0:1789431582.374940:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1789431582.390222:0:333604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431582.390249:0:333604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.390268:0:333604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.390282:0:333604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431582.390290:0:333604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.390297:0:333604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431582.390301:0:333604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431582.390308:0:333604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0d87b58 00010000:00010000:1.0:1789431582.390312:0:333604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.390320:0:333604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431582.390339:0:333604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.390346:0:333604:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=13 00000080:00200008:1.0:1789431582.390355:0:333604:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431582.390360:0:333604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.390373:0:333604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.390381:0:333604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1789431582.390389:0:333604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00000080:00200000:1.0:1789431582.390395:0:333604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.390399:0:333604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.390401:0:333604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431582.390404:0:333604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d87cb8 00010000:00010000:1.0:1789431582.390407:0:333604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 87 lru_type: 1 00010000:00010000:1.0:1789431582.390414:0:333604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000080:00200008:1.0:1789431582.391031:0:333604:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1789431582.391036:0:333604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431582.391044:0:333604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.391052:0:333604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431582.391066:0:333604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431582.391072:0:333604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.391076:0:333604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431582.391078:0:333604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431582.391082:0:333604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0d87cb8 00010000:00010000:1.0:1789431582.391084:0:333604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431582.391091:0:333604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431582.391180:0:333604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431582.391188:0:333604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.391194:0:333604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431582.391202:0:333604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431582.391208:0:333604:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.391213:0:333604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431582.391216:0:333604:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea53 00000080:00010000:1.0:1789431582.391220:0:333604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea53 from it ffffbc42c0d87bb0 00010000:00010000:1.0:1789431582.391223:0:333604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431582.391232:0:333604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:1.0:1789431582.391240:0:333604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 44 00000080:00200008:1.0:1789431582.391247:0:333604:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200080:1.0:1789431582.391262:0:333604:0:(dir.c:2009:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) cmd=c03066f0 arg=7ffce08aa0c0 00000080:00200000:1.0:1789431582.391274:0:333604:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88) name=pjdfstest stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:1.0:1789431582.391294:0:333604:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431582.391337:0:333604:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431582.391345:0:333604:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1789431582.391348:0:333604:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c2600/0x1ffb2ee1bd27ea53 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad345 expref: -99 pid: 331831 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000020:00100000:1.0:1789431582.391369:0:333604:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431582.391381:0:333604:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdca80 x1876350913532416/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431582.391403:0:333604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333604:1876350913532416:192.168.203.128@tcp:36:lfs.0 00000100:00100000:1.0:1789431582.391493:0:333604:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.393081:0:333604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job lfs:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333604:1876350913532416:192.168.203.128@tcp:36:lfs.0 00000080:00200008:1.0:1789431582.393152:0:333604:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), flags 300000 00000080:00200008:2.0:1789431582.435181:0:333064:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.435194:0:333064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431582.435231:0:333064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.435290:0:333064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.435301:0:333064:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333064 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.435310:0:333064:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333064 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.435335:0:333064:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333064:1876350913532544:192.168.203.128@tcp:101:bash.0 00000100:00100000:2.0:1789431582.435419:0:333064:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1789431582.437930:0:333064:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job bash:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333064:1876350913532544:192.168.203.128@tcp:101:bash.0 00010000:00010000:2.0:1789431582.437965:0:333064:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.437979:0:333064:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.437991:0:333064:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438008:0:333064:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438020:0:333064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438027:0:333064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438034:0:333064:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431582.438044:0:333064:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2103480 x1876350913532544/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431582.438057:0:333064:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438080:0:333064:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1789431582.438086:0:333064:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431582.438108:0:333064:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 4096 00000080:00010000:2.0:1789431582.438114:0:333064:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:2.0:1789431582.438119:0:333064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431582.438122:0:333064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c1577cb8 00010000:00010000:2.0:1789431582.438125:0:333064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.438139:0:333064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431582.438185:0:333064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431582.438198:0:333064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.438205:0:333064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.438234:0:333064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431582.438241:0:333064:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.438245:0:333064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431582.438248:0:333064:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:2.0:1789431582.438251:0:333064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c1577b58 00010000:00010000:2.0:1789431582.438254:0:333064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.438261:0:333064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431582.438269:0:333064:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.438280:0:333064:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:2.0:1789431582.438289:0:333064:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.438293:0:333064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431582.438305:0:333064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.438313:0:333064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:2.0:1789431582.438322:0:333064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00000080:00200000:2.0:1789431582.438328:0:333064:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.438332:0:333064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431582.438334:0:333064:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431582.438338:0:333064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1577cb8 00010000:00010000:2.0:1789431582.438341:0:333064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 88 lru_type: 1 00010000:00010000:2.0:1789431582.438348:0:333064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000080:00200000:2.0:1789431582.438366:0:333064:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:2.0:1789431582.438373:0:333064:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.438378:0:333064:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431582.438386:0:333064:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431582.438392:0:333064:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.438396:0:333064:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:2.0:1789431582.438399:0:333064:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:2.0:1789431582.438402:0:333064:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c1577c18 00010000:00010000:2.0:1789431582.438405:0:333064:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431582.438412:0:333064:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431582.438420:0:333064:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.438424:0:333064:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=11 00000080:00200000:2.0:1789431582.438429:0:333064:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 101 00000080:00200000:0.0:1789431582.441417:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441432:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.441444:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441448:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441457:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441461:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441469:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441472:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441480:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441483:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441491:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441494:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441502:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441505:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441512:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441515:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441524:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441527:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:0.0:1789431582.441683:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441688:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.441696:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441700:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441706:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441709:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441716:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441719:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441726:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441728:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441735:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441737:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441744:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441746:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441753:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441756:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.441763:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441766:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:0.0:1789431582.441932:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441938:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.441949:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441952:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441962:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441965:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441971:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441973:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441982:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441986:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.441996:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.441999:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442007:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442010:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442016:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442019:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442026:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442030:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:0.0:1789431582.442163:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442168:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.442177:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442180:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442188:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442191:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442198:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442201:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442209:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442212:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442220:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442223:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442232:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442235:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442243:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442246:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442253:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442257:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:0.0:1789431582.442376:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442382:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.442392:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442396:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442404:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442407:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442415:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442418:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442425:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442428:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442435:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442438:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442444:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442446:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442453:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442455:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442462:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442466:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:0.0:1789431582.442583:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442588:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.442597:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442600:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442606:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442608:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442615:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442617:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442623:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442626:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442632:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442635:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442643:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442646:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442655:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442658:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442666:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442669:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:0.0:1789431582.442781:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442787:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.442796:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442800:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442809:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442812:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442819:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442822:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442830:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442853:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.442866:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442931:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442945:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442949:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442956:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442959:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.442966:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.442970:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:0.0:1789431582.443099:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443104:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.443113:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443116:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.443125:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443128:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.443137:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443140:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.443149:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443152:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.443160:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443164:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.443172:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443175:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.443183:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443185:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.443193:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.443197:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:2.0:1789431582.502155:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913532672:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431582.502258:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789431582.502267:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913532800:192.168.203.128@tcp:400:kworker.0 00010000:00080000:2.0:1789431582.504713:0:2392:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff9811d1bc6300 x1876350913532800/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 0 dl 1789431598 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431582.504731:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913532800:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431582.504744:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913532672:192.168.203.128@tcp:400:kworker.0 00000080:00200000:0.0:1789431582.539501:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539517:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.539531:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539536:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.539545:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539548:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.539558:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539561:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.539570:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539573:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.539581:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539584:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.539592:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539595:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.539603:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539606:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.539614:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.539618:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=librt.so.1, flags=257 00000080:00200000:0.0:1789431582.556763:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.556777:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.proverc, flags=1 00000080:00200000:0.0:1789431582.558183:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.558192:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=.prove, flags=1 00000080:00200000:0.0:1789431582.578159:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.578172:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.578178:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=testrules.yml, flags=1 00000080:00200000:0.0:1789431582.578187:0:333614:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.578191:0:333614:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=t, flags=17 00000080:00200000:3.0:1789431582.663980:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.663994:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.664003:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664006:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664011:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664013:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664017:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664020:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664027:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664030:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664038:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664042:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664051:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664055:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664064:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664067:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664077:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664081:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libperl.so.5.26, flags=257 00000080:00200000:3.0:1789431582.664284:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664290:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.664301:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664304:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664314:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664318:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664328:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664331:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664341:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664344:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664356:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664360:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664371:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664374:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664383:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664386:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664396:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664399:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:3.0:1789431582.664554:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664560:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.664573:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664577:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664587:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664590:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664600:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664604:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664611:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664613:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664618:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664620:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664626:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664628:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664633:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664635:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664641:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664643:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libresolv.so.2, flags=257 00000080:00200000:3.0:1789431582.664795:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664800:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.664810:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664813:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664820:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664823:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664866:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664881:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664897:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664901:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.664915:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664918:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664929:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664932:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664942:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664945:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.664954:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.664958:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431582.665088:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665094:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.665106:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665109:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665120:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665123:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665134:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665138:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665148:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665151:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665161:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665164:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665174:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665177:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665186:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665189:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665199:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665203:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:3.0:1789431582.665329:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665335:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.665344:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665347:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665358:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665361:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665372:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665375:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665388:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665391:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665400:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665402:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665410:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665413:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665422:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665425:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665432:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665434:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypt.so.1, flags=257 00000080:00200000:3.0:1789431582.665544:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665549:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.665557:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665560:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665567:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665570:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665579:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665582:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665590:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665593:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665602:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665605:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665613:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665616:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665624:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665627:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665635:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665638:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libutil.so.1, flags=257 00000080:00200000:3.0:1789431582.665764:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665769:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.665776:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665779:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665787:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665790:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665799:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665802:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665809:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665812:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.665819:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665821:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665829:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665852:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665877:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665882:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.665893:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.665897:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.670337:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670352:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.670366:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670370:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670377:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670380:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670387:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670390:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670398:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670401:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670409:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670412:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670418:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670421:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670428:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670431:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670439:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670443:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431582.670595:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670601:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.670611:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670614:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670624:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670627:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670636:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670639:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670648:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670651:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670658:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670661:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670669:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670672:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670680:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670684:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670691:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670694:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:3.0:1789431582.670804:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670809:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.670816:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670819:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670826:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670829:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670860:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670864:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670886:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670889:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.670898:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670901:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670907:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670909:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670914:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670915:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.670920:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.670922:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.672538:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.672562:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672579:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.672592:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.672597:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672603:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.672605:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.672611:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbb58 00010000:00010000:3.0:1789431582.672613:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.672620:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.672632:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.672636:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431582.672642:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.672646:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.672654:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672659:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:3.0:1789431582.672664:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00000080:00200000:3.0:1789431582.672668:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672670:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.672672:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.672673:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.672675:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 89 lru_type: 1 00010000:00010000:3.0:1789431582.672679:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000080:00200000:3.0:1789431582.672688:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.672691:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.672693:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.672697:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672700:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:3.0:1789431582.672708:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00000080:00200000:3.0:1789431582.672714:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672718:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.672720:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.672724:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.672727:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 90 lru_type: 1 00010000:00010000:3.0:1789431582.672734:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200008:3.0:1789431582.672754:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.672758:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.672764:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672770:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.672778:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.672785:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672789:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.672792:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.672795:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.672797:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.672802:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.672813:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.672817:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672822:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.672829:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.672857:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672861:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.672864:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.672878:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbb58 00010000:00010000:3.0:1789431582.672881:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.672889:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.672896:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.672901:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431582.672906:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.672910:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.672918:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.672925:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:3.0:1789431582.672930:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00000080:00200000:3.0:1789431582.672934:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.672937:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.672939:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.672941:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.672943:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 91 lru_type: 1 00010000:00010000:3.0:1789431582.672948:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:1.0:1789431582.674508:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.674520:0:333616:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.674526:0:333616:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.674540:0:333616:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.674552:0:333616:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1789431582.674563:0:333616:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00000080:00200000:1.0:1789431582.674568:0:333616:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.674573:0:333616:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.674575:0:333616:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431582.674578:0:333616:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c0d9bcb8 00010000:00010000:1.0:1789431582.674581:0:333616:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 92 lru_type: 1 00010000:00010000:1.0:1789431582.674586:0:333616:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:1.0:1789431582.675629:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675638:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.675649:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675653:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.675660:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675664:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.675671:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675673:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.675677:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675679:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.675684:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675686:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.675691:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675692:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.675696:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675698:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.675702:0:333616:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.675705:0:333616:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.681443:0:333619:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.681457:0:333619:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.681464:0:333619:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.681488:0:333619:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.681507:0:333619:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1789431582.681522:0:333619:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00000080:00200000:1.0:1789431582.681531:0:333619:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.681537:0:333619:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.681541:0:333619:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431582.681547:0:333619:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431582.681551:0:333619:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 93 lru_type: 1 00010000:00010000:1.0:1789431582.681560:0:333619:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:2.0:1789431582.683166:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683175:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.683181:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683184:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.683188:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683190:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.683194:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683196:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.683200:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683202:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.683207:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683208:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.683212:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683214:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.683219:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683221:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.683225:0:333620:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.683227:0:333620:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.687276:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.687294:0:333622:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.687302:0:333622:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.687327:0:333622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.687345:0:333622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:3.0:1789431582.687360:0:333622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00000080:00200000:3.0:1789431582.687369:0:333622:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.687376:0:333622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.687380:0:333622:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.687386:0:333622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c154bcb8 00010000:00010000:3.0:1789431582.687391:0:333622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 94 lru_type: 1 00010000:00010000:3.0:1789431582.687400:0:333622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200000:0.0:1789431582.687727:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431582.687737:0:333623:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431582.687741:0:333623:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431582.687752:0:333623:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431582.687762:0:333623:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:0.0:1789431582.687772:0:333623:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00000080:00200000:0.0:1789431582.687778:0:333623:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431582.687783:0:333623:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431582.687786:0:333623:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431582.687790:0:333623:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1f23cb8 00010000:00010000:0.0:1789431582.687793:0:333623:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 95 lru_type: 1 00010000:00010000:0.0:1789431582.687802:0:333623:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:0.0:1789431582.688940:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688948:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.688955:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688957:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.688963:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688965:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.688969:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688971:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.688976:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688978:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.688982:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688984:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.688989:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688990:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.688994:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.688996:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.689000:0:333623:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.689002:0:333623:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.689328:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689334:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.689346:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689349:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.689357:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689360:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.689369:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689372:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.689382:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689386:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.689398:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689401:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.689417:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689420:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.689432:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689435:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.689443:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.689447:0:333622:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.690267:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.690274:0:333622:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.690279:0:333622:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.690296:0:333622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.690310:0:333622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:3.0:1789431582.690323:0:333622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00000080:00200000:3.0:1789431582.690331:0:333622:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.690337:0:333622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.690340:0:333622:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.690345:0:333622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c154bcb8 00010000:00010000:3.0:1789431582.690349:0:333622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 96 lru_type: 1 00010000:00010000:3.0:1789431582.690358:0:333622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000080:00200000:3.0:1789431582.690381:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.690385:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 44 00000080:00200008:3.0:1789431582.690391:0:333622:0:(file.c:1090:ll_file_open()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100400 00000080:00200008:3.0:1789431582.690407:0:333622:0:(file.c:479:ll_file_release()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), flags 100000 00010000:00010000:3.0:1789431582.690420:0:333622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1789431582.690561:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.690570:0:333624:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.690574:0:333624:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431582.690585:0:333624:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.690593:0:333624:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:2.0:1789431582.690603:0:333624:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00000080:00200000:2.0:1789431582.690609:0:333624:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.690613:0:333624:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431582.690615:0:333624:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431582.690621:0:333624:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1593cb8 00010000:00010000:2.0:1789431582.690624:0:333624:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 97 lru_type: 1 00010000:00010000:2.0:1789431582.690633:0:333624:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000080:00200008:3.0:1789431582.691174:0:333622:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.691180:0:333622:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.691189:0:333622:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.691197:0:333622:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.691207:0:333622:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.691214:0:333622:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.691218:0:333622:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.691220:0:333622:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.691225:0:333622:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c154bcb8 00010000:00010000:3.0:1789431582.691228:0:333622:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.691236:0:333622:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431582.691255:0:333622:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.691261:0:333622:0:(llite_lib.c:2904:ll_statfs()) VFS Op:sb=lustre (ffff9811d1e45800) pjdfstest proj=0 00000080:00000004:3.0:1789431582.691266:0:333622:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff9811d1e45800: age 3948, max_age 3956 00800000:00000004:3.0:1789431582.691276:0:333622:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: age 3948, max_age 3956 00000100:00100000:3.0:1789431582.691315:0:333622:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333622:1876350913532928:192.168.203.128@tcp:41:df.0 00000100:00100000:3.0:1789431582.691392:0:333622:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cdaa80 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.692353:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692362:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.692372:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692375:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692383:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692386:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692393:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692395:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692403:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692406:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692412:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692415:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692419:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692421:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692425:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692427:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692431:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692434:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431582.692549:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692553:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.692558:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692560:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692564:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692566:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692570:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692572:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692576:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692577:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692582:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692583:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692588:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692589:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692594:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692595:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692599:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692602:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431582.692691:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692695:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.692700:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692702:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692706:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692708:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692712:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692714:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692719:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692720:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692725:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692726:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692731:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692732:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692737:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692739:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692746:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692749:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431582.692915:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692920:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.692929:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692932:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692942:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692946:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692955:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692959:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692970:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692974:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.692983:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692986:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.692994:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.692997:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693008:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693011:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693020:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693024:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431582.693141:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693147:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.693154:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693157:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693164:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693167:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693174:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693177:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693185:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693189:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693197:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693200:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693208:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693211:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693219:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693222:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693230:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693233:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:1.0:1789431582.693346:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693351:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.693360:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693363:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693370:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693373:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693381:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693384:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693391:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693394:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693402:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693405:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693413:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693416:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693423:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693426:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693433:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693437:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:1.0:1789431582.693548:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693553:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.693563:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693566:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693575:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693577:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693584:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693587:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693593:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693596:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693603:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693606:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693613:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693616:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693624:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693626:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.693633:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693637:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.693786:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693791:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.693799:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693802:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693810:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693813:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.693820:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.693822:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.694000:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.694007:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.694022:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.694026:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.694038:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.694041:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.694055:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.694059:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.694076:0:333624:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.694081:0:333624:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000100:00100000:3.0:1789431582.695030:0:333622:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job df:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333622:1876350913532928:192.168.203.128@tcp:41:df.0 00800000:00000004:3.0:1789431582.695059:0:333622:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff9811d1e45800: update ffff9811c6a537d8 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431582.695063:0:333622:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff9811d1e45800: update ffff9811c6a50458 cache blocks 1802738/1916558 objects 523571/524106 00000080:00000004:3.0:1789431582.695067:0:333622:0:(llite_lib.c:2756:ll_statfs_internal()) MDC blocks 1802738/1916558 objects 523571/524106 00000080:00200000:3.0:1789431582.704828:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.704933:0:333627:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.704941:0:333627:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.704973:0:333627:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.704991:0:333627:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:3.0:1789431582.705006:0:333627:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00000080:00200000:3.0:1789431582.705014:0:333627:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.705022:0:333627:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.705025:0:333627:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.705032:0:333627:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1f23cb8 00010000:00010000:3.0:1789431582.705036:0:333627:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 98 lru_type: 1 00010000:00010000:3.0:1789431582.705045:0:333627:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000080:00200000:1.0:1789431582.705738:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.705749:0:333628:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.705755:0:333628:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.705768:0:333628:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.705778:0:333628:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1789431582.705789:0:333628:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00000080:00200000:1.0:1789431582.705797:0:333628:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.705804:0:333628:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.705807:0:333628:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431582.705812:0:333628:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c163fcb8 00010000:00010000:1.0:1789431582.705816:0:333628:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 99 lru_type: 1 00010000:00010000:1.0:1789431582.705825:0:333628:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000080:00200000:2.0:1789431582.706282:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.706292:0:333629:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.706297:0:333629:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431582.706310:0:333629:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.706319:0:333629:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:2.0:1789431582.706331:0:333629:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00000080:00200000:2.0:1789431582.706338:0:333629:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.706343:0:333629:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431582.706346:0:333629:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431582.706350:0:333629:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1677cb8 00010000:00010000:2.0:1789431582.706353:0:333629:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 100 lru_type: 1 00010000:00010000:2.0:1789431582.706361:0:333629:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000080:00200000:3.0:1789431582.706744:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706752:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.706767:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706772:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.706785:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706788:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.706801:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706804:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.706817:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706821:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.706912:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706916:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.706928:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706930:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.706939:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706941:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.706947:0:333627:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.706950:0:333627:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.707397:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707404:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.707418:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707422:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707433:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707437:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707453:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707457:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707467:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707471:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707480:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707484:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.707495:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707498:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.707509:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707512:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.707526:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707530:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:1.0:1789431582.707779:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707784:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.707802:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707806:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707827:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707857:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707890:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707895:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707919:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707923:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.707946:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707951:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.707973:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.707977:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.707996:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708000:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.708016:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708022:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:2.0:1789431582.708031:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708038:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.708054:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708059:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708071:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708074:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708085:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708089:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708100:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708104:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708117:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708120:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708130:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708133:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708146:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708149:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708159:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708163:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431582.708227:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708233:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.708248:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708251:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708263:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708265:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708276:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708280:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708294:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708298:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708313:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708317:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.708340:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708343:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708345:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708352:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.708366:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708371:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708373:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708380:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708392:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708398:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431582.708400:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708406:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708427:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708431:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708444:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708447:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708457:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708460:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708470:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708473:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708485:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708488:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708499:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708502:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431582.708582:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708588:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.708599:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708602:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708616:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708619:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708634:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708638:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708656:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708660:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708666:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708673:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.708676:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708682:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708690:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708695:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708700:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708705:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708712:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708718:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708722:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708728:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708734:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708739:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708745:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708751:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431582.708755:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708761:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.708778:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708781:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708794:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708796:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708806:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708810:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.708821:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.708825:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431582.708933:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708939:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.708953:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708956:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708967:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708970:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708981:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708984:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.708994:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.708997:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709013:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709018:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709019:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709027:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.709036:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709041:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709045:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709050:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709057:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709062:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709066:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709070:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709078:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709083:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431582.709089:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709094:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709110:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709113:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709125:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709128:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709139:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709142:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709154:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709157:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709167:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709171:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:1.0:1789431582.709244:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709250:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.709263:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709267:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709281:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709284:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709304:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709308:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709315:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709322:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.709327:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709332:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709341:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709346:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709350:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709355:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709363:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709368:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709372:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709378:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709384:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709390:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709395:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709401:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709406:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709411:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.709419:0:333628:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.709425:0:333628:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.709428:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709433:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709450:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709453:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709464:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709467:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709479:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709482:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431582.709610:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709615:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.709624:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709627:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709635:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709638:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709648:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709651:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709661:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709664:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709673:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709676:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709684:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709688:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709697:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709700:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709708:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709712:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431582.709881:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709887:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.709904:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709908:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709921:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709924:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709939:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709943:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709952:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709956:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.709966:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709970:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709980:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709982:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.709993:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.709996:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.710004:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710007:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.710186:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710192:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.710202:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710205:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.710214:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710217:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.710225:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710228:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.710239:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710242:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.710252:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710255:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.710264:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710268:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.710276:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710279:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.710288:0:333629:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.710292:0:333629:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:3.0:1789431582.722407:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.722425:0:333632:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.722432:0:333632:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.722458:0:333632:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.722476:0:333632:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:3.0:1789431582.722491:0:333632:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00000080:00200000:3.0:1789431582.722499:0:333632:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.722507:0:333632:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.722511:0:333632:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:3.0:1789431582.722518:0:333632:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c174bcb8 00010000:00010000:3.0:1789431582.722522:0:333632:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 101 lru_type: 1 00010000:00010000:3.0:1789431582.722532:0:333632:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000080:00200000:1.0:1789431582.723151:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.723160:0:333633:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.723165:0:333633:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.723177:0:333633:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.723186:0:333633:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1789431582.723197:0:333633:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00000080:00200000:1.0:1789431582.723203:0:333633:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.723208:0:333633:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.723211:0:333633:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:1.0:1789431582.723215:0:333633:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c1753cb8 00010000:00010000:1.0:1789431582.723218:0:333633:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 102 lru_type: 1 00010000:00010000:1.0:1789431582.723226:0:333633:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000080:00200000:2.0:1789431582.723694:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.723703:0:333634:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.723708:0:333634:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431582.723720:0:333634:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.723729:0:333634:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:2.0:1789431582.723740:0:333634:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00000080:00200000:2.0:1789431582.723749:0:333634:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.723753:0:333634:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:2.0:1789431582.723756:0:333634:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:2.0:1789431582.723761:0:333634:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c178bcb8 00010000:00010000:2.0:1789431582.723764:0:333634:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 103 lru_type: 1 00010000:00010000:2.0:1789431582.723770:0:333634:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000080:00200000:3.0:1789431582.724199:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724207:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.724222:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724226:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.724239:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724243:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.724254:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724257:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.724269:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724273:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.724288:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724292:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.724303:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724305:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.724317:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724320:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.724332:0:333632:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.724336:0:333632:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.724653:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724658:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.724668:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724670:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.724679:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724681:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.724691:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724694:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.724706:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724709:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.724722:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724725:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.724737:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724740:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.724752:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724755:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.724767:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.724771:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libssl.so.1.1, flags=257 00000080:00200000:1.0:1789431582.725043:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725049:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.725063:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725067:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725080:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725083:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725097:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725100:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725118:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725121:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725140:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725144:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725163:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725166:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725182:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725186:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725201:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725206:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libcrypto.so.1.1, flags=257 00000080:00200000:1.0:1789431582.725405:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725410:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.725417:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725425:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.725428:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725432:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.725448:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725450:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725456:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725457:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.725481:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725482:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725488:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725490:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725515:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725515:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725525:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.725525:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725548:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725552:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725553:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725561:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725569:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725575:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725579:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725585:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725590:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725595:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725605:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725610:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725613:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725620:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libz.so.1, flags=257 00000080:00200000:2.0:1789431582.725632:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725637:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725658:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725663:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libsigsegv.so.2, flags=257 00000080:00200000:1.0:1789431582.725797:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725803:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.725814:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725818:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725855:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725860:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725891:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725896:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.725898:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725906:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.725917:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725922:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.725926:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725932:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725938:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725942:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725953:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725957:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725960:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.725961:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.725980:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.725984:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.725987:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.725993:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726001:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726007:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431582.726012:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726018:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726034:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726038:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726050:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726053:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726065:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726068:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726080:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726084:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libreadline.so.7, flags=257 00000080:00200000:1.0:1789431582.726141:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726147:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.726159:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726162:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726169:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726171:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726178:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726180:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726185:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726187:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726191:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726194:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726203:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726206:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726215:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726218:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726230:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726234:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:2.0:1789431582.726264:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726270:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.726286:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726289:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726303:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726307:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726319:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726322:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726335:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726339:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726350:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726353:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726365:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726368:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726378:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726382:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726397:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726402:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libmpfr.so.4, flags=257 00000080:00200000:1.0:1789431582.726412:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726418:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.726433:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726436:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726446:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726450:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726461:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726464:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726475:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726478:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.726491:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726495:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726507:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726511:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726522:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726524:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.726535:0:333633:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.726538:0:333633:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.726570:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726577:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.726594:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726598:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726610:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726613:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726627:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726630:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726643:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726647:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.726657:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726661:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726674:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726677:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726689:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726692:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.726701:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726705:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libgmp.so.10, flags=257 00000080:00200000:2.0:1789431582.726854:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.726861:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.727088:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727093:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727106:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727110:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727121:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727125:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727135:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727139:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727151:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727155:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727167:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727171:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727180:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727184:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727192:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727195:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libdl.so.2, flags=257 00000080:00200000:2.0:1789431582.727332:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727338:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.727350:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727354:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727366:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727369:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727381:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727384:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727396:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727399:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727408:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727411:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727422:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727425:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727435:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727438:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727447:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727451:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libm.so.6, flags=257 00000080:00200000:2.0:1789431582.727585:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727590:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.727602:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727606:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727616:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727620:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727629:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727632:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727641:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727644:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727654:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727657:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727668:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727671:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727683:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727686:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727698:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727701:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.727905:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727911:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.727922:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727926:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727935:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727938:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727947:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727950:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727959:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727962:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.727971:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727974:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727982:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727985:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.727996:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.727999:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.728009:0:333634:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.728013:0:333634:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libtinfo.so.6, flags=257 00000080:00200000:0.0:1789431582.736906:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:0.0:1789431582.736927:0:333637:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431582.736934:0:333637:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:0.0:1789431582.736958:0:333637:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431582.736975:0:333637:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:0.0:1789431582.736991:0:333637:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00000080:00200000:0.0:1789431582.736998:0:333637:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431582.737006:0:333637:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:0.0:1789431582.737011:0:333637:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27e996 00000080:00010000:0.0:1789431582.737020:0:333637:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27e996 from it ffffbc42c154bcb8 00010000:00010000:0.0:1789431582.737026:0:333637:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 104 lru_type: 1 00010000:00010000:0.0:1789431582.737038:0:333637:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000080:00200000:2.0:1789431582.737802:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737811:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.737826:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737848:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.737886:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737891:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.737902:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737906:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.737916:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737920:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.737930:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737934:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.737943:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737946:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.737954:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737957:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.737965:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.737969:0:333636:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431582.738253:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738259:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.738271:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738275:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.738283:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738286:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.738294:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738297:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.738307:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738310:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.738318:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738321:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.738330:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738333:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.738342:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738345:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.738355:0:333637:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.738358:0:333637:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.738475:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.738485:0:333636:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=1536 00000080:00200000:2.0:1789431582.738488:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431582.738491:0:333636:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 3 00000080:00200000:2.0:1789431582.738495:0:333636:0:(namei.c:2165:do_mkdir()) VFS Op:name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8) 00010000:00010000:2.0:1789431582.738520:0:333636:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:2.0:1789431582.738541:0:333636:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1789431582.738651:0:333636:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00010000:00010000:2.0:1789431582.738657:0:333636:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431582.738659:0:333636:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3200/0x1ffb2ee1bd27e996 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad123 expref: -99 pid: 327622 timeout: 0 lvb_type: 0 lru_score: 105 lru_type: 1 00000020:00100000:2.0:1789431582.738671:0:333636:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1789431582.738680:0:333636:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2103480 x1876350913533184/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 592/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431582.738697:0:333636:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333636:1876350913533184:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431582.738770:0:333636:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.741375:0:333636:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103480 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333636:1876350913533184:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431582.741413:0:333636:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00200000:3.0:1789431582.741419:0:333636:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a7:0x0], updating i_size 4096 00000080:00200000:2.0:1789431582.744885:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.744902:0:333639:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.744908:0:333639:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:2.0:1789431582.744925:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.744938:0:333639:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1 (0 0) 00010000:00010000:2.0:1789431582.744984:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.744993:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.745002:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.745026:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533312:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431582.745105:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.746777:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533312:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431582.746797:0:333639:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746809:0:333639:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746820:0:333639:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746879:0:333639:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746893:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746902:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746909:0:333639:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.746920:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2103800 x1876350913533312/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/688 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.746932:0:333639:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.746943:0:333639:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1 (0 0) 00000080:00200000:3.0:1789431582.746953:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431582.746976:0:333639:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00010000:3.0:1789431582.746981:0:333639:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27ea76 00000080:00010000:3.0:1789431582.746987:0:333639:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.746990:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174bcb8 00010000:00010000:3.0:1789431582.746993:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.747015:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.748243:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.748255:0:333639:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431582.748260:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431582.748279:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.748320:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.748328:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.748336:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.748356:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533440:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.748422:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.750310:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533440:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.750328:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750339:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750346:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750358:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750367:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750373:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.750377:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750384:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750391:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750397:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.750403:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750409:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27ea7d lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.750424:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913533440/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.750437:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.750457:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.750467:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431582.750476:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b970 00010000:00010000:3.0:1789431582.750480:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.750488:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.750540:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.750551:0:333639:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431582.750555:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431582.750565:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.750588:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750595:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.750601:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.750617:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533568:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.750662:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.751763:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533568:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.751776:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751785:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751792:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751799:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751805:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751811:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.751814:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751819:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751825:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751853:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.751860:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.751880:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ea84 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.751893:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913533568/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.751905:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.751919:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.751929:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431582.751936:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b970 00010000:00010000:3.0:1789431582.751940:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.751948:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.751984:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.751992:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.752012:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.752016:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.752026:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.752029:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.752038:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.752046:0:333639:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=17 00000080:00200000:3.0:1789431582.752050:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=0 00000002:00010000:3.0:1789431582.752060:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.752082:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.752089:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.752096:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.752116:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533696:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.752162:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.753534:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533696:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.753545:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753559:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753565:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753570:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753576:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753582:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.753586:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753592:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753599:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753605:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.753609:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753615:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27ea8b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.753628:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913533696/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.753639:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.753655:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.753664:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431582.753672:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b970 00010000:00010000:3.0:1789431582.753675:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.753682:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.753731:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.753736:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.753744:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.753747:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.753754:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.753762:0:333639:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c53ebe00, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.753767:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431582.753778:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.753806:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.753812:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.753821:0:333639:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.753827:0:333639:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913533952/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.753880:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.753896:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533952:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.754030:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.755644:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913533952:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.755661:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755671:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755678:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755692:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755700:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755708:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.755711:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755719:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755726:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755732:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.755743:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.755748:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27ea92 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.755760:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913533952/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.755773:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.755793:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.755803:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431582.755812:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b9a0 00010000:00010000:3.0:1789431582.755815:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.755828:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.756166:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756174:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.756186:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756190:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.756198:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756202:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.756211:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756214:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.756219:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756221:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.756226:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756228:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.756232:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756234:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.756239:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756240:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.756245:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.756251:0:333639:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811c2c47980, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.756257:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431582.756270:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.756305:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.756314:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.756327:0:333639:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.756333:0:333639:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913534208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.756342:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.756359:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913534208:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.756482:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.758088:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913534208:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.758104:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758114:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758119:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758124:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758129:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758132:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.758136:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758143:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758150:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758156:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.758161:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758167:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27ea99 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.758180:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913534208/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.758193:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.758212:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.758222:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431582.758229:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b9a0 00010000:00010000:3.0:1789431582.758232:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.758239:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.758546:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758554:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.758564:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758567:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.758572:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758574:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.758578:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758580:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.758584:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758586:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.758590:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758592:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.758597:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758599:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.758606:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758609:0:333639:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.758616:0:333639:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.758626:0:333639:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), file ffff9811d30b1200, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.758631:0:333639:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=open 00000002:00010000:3.0:1789431582.758645:0:333639:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.758675:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.758680:0:333639:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.758688:0:333639:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.758693:0:333639:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913534464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.758703:0:333639:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.758720:0:333639:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913534464:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.758866:0:333639:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.760135:0:333639:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333639:1876350913534464:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.760148:0:333639:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760158:0:333639:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760164:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760171:0:333639:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760179:0:333639:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760184:0:333639:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.760188:0:333639:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760194:0:333639:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760200:0:333639:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x20e400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760207:0:333639:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.760211:0:333639:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.760218:0:333639:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eaa0 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333639 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.760229:0:333639:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913534464/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.760247:0:333639:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.760260:0:333639:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.760267:0:333639:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:3.0:1789431582.760272:0:333639:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c174b9a0 00010000:00010000:3.0:1789431582.760275:0:333639:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.760279:0:333639:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431582.763349:0:333640:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.763365:0:333640:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.763383:0:333640:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:1.0:1789431582.763403:0:333640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.763419:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.763434:0:333640:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431582.763442:0:333640:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.763449:0:333640:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.763452:0:333640:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27ea76 00000080:00010000:1.0:1789431582.763459:0:333640:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431582.763463:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.763473:0:333640:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431582.763501:0:333640:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:1.0:1789431582.763509:0:333640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.763516:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.763525:0:333640:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1789431582.763537:0:333640:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.763541:0:333640:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:1.0:1789431582.763543:0:333640:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:1.0:1789431582.763547:0:333640:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c154bb58 00010000:00010000:1.0:1789431582.763550:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:1.0:1789431582.763558:0:333640:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431582.763567:0:333640:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.763579:0:333640:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), flags=9 00000080:00200000:1.0:1789431582.763583:0:333640:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest, dir=[0x200000007:0x1:0x0](ffff9811d8113a88), intent=getattr 00000002:00010000:1.0:1789431582.763595:0:333640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1789431582.763634:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333640 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.763641:0:333640:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333640 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.763651:0:333640:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333640 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.763674:0:333640:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333640:1876350913534592:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431582.763757:0:333640:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.765502:0:333640:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333640:1876350913534592:192.168.203.128@tcp:101:sh.0 00010000:00010000:1.0:1789431582.765524:0:333640:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765535:0:333640:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765543:0:333640:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765551:0:333640:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765559:0:333640:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765569:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765576:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765583:0:333640:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.765595:0:333640:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdca80 x1876350913534592/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/688 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431582.765607:0:333640:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765617:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431582.765628:0:333640:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431582.765635:0:333640:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765642:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765649:0:333640:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.765669:0:333640:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1789431582.765674:0:333640:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.765696:0:333640:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1789431582.765699:0:333640:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x1:0x0](ffff9811d82763c8): rc = 0 00010000:00010000:0.0:1789431582.765706:0:6161:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00000080:00010000:1.0:1789431582.765706:0:333640:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27ea76 00010000:00010000:0.0:1789431582.765717:0:6161:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1789431582.765722:0:333640:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.765728:0:333640:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c154bb20 00010000:00010000:0.0:1789431582.765734:0:6161:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.765735:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1789431582.765746:0:333640:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:0.0:1789431582.765748:0:6161:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.765764:0:6161:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.765778:0:6161:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:1.0:1789431582.765785:0:333640:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.765792:0:333640:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00010000:00010000:0.0:1789431582.765796:0:6161:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c0030800/0x1ffb2ee1bd25cbcc lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f54079537662 expref: -99 pid: 17166 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1789431582.765807:0:333640:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431582.765809:0:6161:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431582.765817:0:333640:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1789431582.765828:0:333640:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:1.0:1789431582.765860:0:333640:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.765865:0:333640:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:1.0:1789431582.765880:0:333640:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27ea76 00010000:00010000:0.0:1789431582.765883:0:6161:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 2/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1789431582.765886:0:333640:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c154bcb8 00010000:00010000:1.0:1789431582.765891:0:333640:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:0.0:1789431582.765895:0:6161:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1789431582.765905:0:333640:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1789431582.765919:0:6161:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.765929:0:6161:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eaa7 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x13 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad44f expref: -99 pid: 333640 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.765940:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431582.765941:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431582.765943:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431582.765955:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913534720:192.168.203.128@tcp:103:ldlm_bl.0 00000080:00200008:3.0:1789431582.766966:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff9811d8113a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.766974:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.766985:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.766996:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431582.767005:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431582.767010:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.767013:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.767015:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.767019:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.767021:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431582.767026:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431582.767039:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.767043:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.767048:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431582.767056:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:3.0:1789431582.767061:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.767065:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.767067:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.767070:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbb58 00010000:00010000:3.0:1789431582.767073:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:3.0:1789431582.767080:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431582.767089:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.767094:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=9 00000080:00200008:3.0:1789431582.767100:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest:[0x200000402:0x1:0x0](ffff9811d82763c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.767103:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8),name=pjdfstest 00000002:00010000:3.0:1789431582.767110:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.767114:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1789431582.767120:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1789431582.767124:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.767126:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) 00000080:00010000:3.0:1789431582.767127:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x1:0x0](ffff9811d82763c8) for lock 0x1ffb2ee1bd27ea76 00000080:00010000:3.0:1789431582.767130:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea76 from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.767133:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1789431582.767140:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63e00/0x1ffb2ee1bd27ea76 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad3b5 expref: -99 pid: 333639 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1789431582.767155:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.767161:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.767168:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431582.767176:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431582.767183:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.767186:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.767189:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.767192:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbb58 00010000:00010000:3.0:1789431582.767195:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431582.767202:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1789431582.767210:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.767214:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=25 00000080:00200000:3.0:1789431582.767218:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.767229:0:333615:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), flags=9 00000080:00200000:3.0:1789431582.767233:0:333615:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f, dir=[0x200000402:0x1:0x0](ffff9811d82763c8), intent=getattr 00000002:00010000:3.0:1789431582.767240:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f,[0x0:0x0:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431582.767268:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333615 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.767276:0:333615:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333615 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.767284:0:333615:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333615 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.767301:0:333615:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333615:1876350913534848:192.168.203.128@tcp:101:sh.0 00000100:00100000:3.0:1789431582.767344:0:333615:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.767404:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913534720:192.168.203.128@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1789431582.768770:0:333615:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333615:1876350913534848:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431582.768798:0:333615:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: --/CR res: [0x200000402:0x1:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768817:0:333615:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768824:0:333615:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768881:0:333615:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768895:0:333615:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768910:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768920:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768929:0:333615:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.768947:0:333615:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4a80 x1876350913534848/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.768960:0:333615:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.768971:0:333615:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1703 (0 0) 00000080:00200000:3.0:1789431582.768977:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431582.769000:0:333615:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a7:0x0], updating i_size 4096 00000080:00200000:3.0:1789431582.769004:0:333615:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x6a7:0x0](ffff9811c4823a88): rc = 0 00000080:00010000:3.0:1789431582.769010:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eaae 00000080:00010000:3.0:1789431582.769028:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.769032:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0ddbb20 00010000:00010000:3.0:1789431582.769035:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.769044:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1789431582.769081:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.769088:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:3.0:1789431582.769110:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.769119:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.769130:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.769138:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.769142:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.769145:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eaae 00000080:00010000:3.0:1789431582.769149:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0ddbcb8 00010000:00010000:3.0:1789431582.769152:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.769159:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.769312:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88),name=/ 00000002:00010000:3.0:1789431582.769322:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.769329:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:3.0:1789431582.769337:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1789431582.769343:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.769347:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) 00000080:00010000:3.0:1789431582.769349:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff9811d8113a88) for lock 0x1ffb2ee1bd27ea6f 00000080:00010000:3.0:1789431582.769353:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ea6f from it ffffbc42c0ddbc18 00010000:00010000:3.0:1789431582.769356:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:3.0:1789431582.769364:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63800/0x1ffb2ee1bd27ea6f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad392 expref: -99 pid: 333064 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:3.0:1789431582.769372:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff9811d8113a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.769379:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest, flags=27 00000080:00200000:3.0:1789431582.769383:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x1:0x0](ffff9811d82763c8), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.769387:0:333615:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f, flags=11 00000080:00200000:3.0:1789431582.769392:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 101 00000080:00200000:1.0:1789431582.772368:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.772387:0:333643:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.772394:0:333643:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:1.0:1789431582.772416:0:333643:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.772431:0:333643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431582.772443:0:333643:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431582.772451:0:333643:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.772461:0:333643:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.772466:0:333643:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eaae 00000080:00010000:1.0:1789431582.772474:0:333643:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c1bebcb8 00010000:00010000:1.0:1789431582.772478:0:333643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1789431582.772488:0:333643:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.773217:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.773236:0:333642:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.773241:0:333642:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.773261:0:333642:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.773325:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.773335:0:333642:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.773346:0:333642:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.773377:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913534976:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431582.773480:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.774015:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:3.0:1789431582.775561:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913534976:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.775582:0:333642:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775600:0:333642:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775614:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775631:0:333642:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775641:0:333642:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775648:0:333642:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.775652:0:333642:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775660:0:333642:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775667:0:333642:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775675:0:333642:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.775682:0:333642:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775690:0:333642:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eab5 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.775707:0:333642:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913534976/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.775719:0:333642:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.775760:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.775770:0:333642:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431582.775777:0:333642:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0dd3970 00010000:00010000:3.0:1789431582.775780:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.775789:0:333642:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.775902:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.775918:0:333642:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.775923:0:333642:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.775939:0:333642:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.775989:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.775997:0:333642:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.776006:0:333642:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.776030:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535104:192.168.203.128@tcp:101:pjdfstest.0 00000080:00200000:1.0:1789431582.776079:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000100:00100000:3.0:1789431582.776086:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.776111:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:3.0:1789431582.778008:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535104:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.778028:0:333642:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778041:0:333642:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778048:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778054:0:333642:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778062:0:333642:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778068:0:333642:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.778072:0:333642:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778079:0:333642:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778086:0:333642:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778092:0:333642:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.778097:0:333642:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.778104:0:333642:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eabc lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.778126:0:333642:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913535104/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.778140:0:333642:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.778180:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.778192:0:333642:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431582.778201:0:333642:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0dd3970 00010000:00010000:3.0:1789431582.778205:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.778212:0:333642:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.778262:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.778268:0:333642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.778275:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.778278:0:333642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.778286:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.778288:0:333642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.778301:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.778301:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.778317:0:333642:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.778321:0:333642:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000080:00200000:1.0:1789431582.778335:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000002:00010000:3.0:1789431582.778336:0:333642:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00000080:00200000:1.0:1789431582.778349:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.778365:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00010000:00010000:3.0:1789431582.778366:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.778370:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00010000:00010000:3.0:1789431582.778375:0:333642:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.778385:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.778389:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00010000:00010000:3.0:1789431582.778391:0:333642:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.778399:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:3.0:1789431582.778415:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535232:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431582.778479:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.780039:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535232:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.780057:0:333642:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780068:0:333642:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780075:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780082:0:333642:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780090:0:333642:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780097:0:333642:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.780101:0:333642:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780107:0:333642:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780114:0:333642:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780120:0:333642:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.780125:0:333642:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.780131:0:333642:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eac3 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.780144:0:333642:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913535232/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.780156:0:333642:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.780186:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.780199:0:333642:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431582.780207:0:333642:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0dd3970 00010000:00010000:3.0:1789431582.780210:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.780218:0:333642:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1789431582.780240:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.780275:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.780276:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.780285:0:333642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.780286:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.780310:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.780311:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.780318:0:333642:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.780319:0:333643:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.780341:0:333643:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.780343:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.780353:0:333643:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c388fc80, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431582.780363:0:333643:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:1.0:1789431582.780379:0:333643:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431582.780435:0:333643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.780445:0:333643:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431582.780466:0:333643:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431582.780482:0:333643:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdca80 x1876350913535488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431582.780490:0:333643:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.780509:0:333643:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333643:1876350913535488:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.780684:0:333643:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.782493:0:333643:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333643:1876350913535488:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.782511:0:333643:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782521:0:333643:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782528:0:333643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782543:0:333643:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782551:0:333643:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782561:0:333643:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f000 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.782565:0:333643:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782572:0:333643:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782579:0:333643:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782586:0:333643:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.782592:0:333643:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.782598:0:333643:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eaca lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333643 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.782613:0:333643:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdca80 x1876350913535488/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.782624:0:333643:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431582.782661:0:333643:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1789431582.782673:0:333643:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.782681:0:333642:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c5713080, open_flags 8000, mode 0 opened 0 00000080:00010000:1.0:1789431582.782682:0:333643:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c1beb9a0 00010000:00010000:1.0:1789431582.782688:0:333643:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.782694:0:333642:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00010000:00010000:1.0:1789431582.782711:0:333643:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000002:00010000:3.0:1789431582.782716:0:333642:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.782761:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.782770:0:333642:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.782780:0:333642:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.782788:0:333642:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913535744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.782798:0:333642:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.782817:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535744:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431582.783018:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.784924:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913535744:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.784946:0:333642:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.784959:0:333642:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.784977:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.784985:0:333642:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.784994:0:333642:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785001:0:333642:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.785004:0:333642:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785011:0:333642:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785018:0:333642:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785025:0:333642:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.785031:0:333642:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785037:0:333642:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27ead1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.785056:0:333642:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4a80 x1876350913535744/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.785076:0:333642:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.785103:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431582.785115:0:333642:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00010000:3.0:1789431582.785124:0:333642:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eaae from it ffffbc42c0dd39a0 00010000:00010000:3.0:1789431582.785128:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431582.785136:0:333642:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1789431582.785764:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.785782:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 3 00000080:00200000:3.0:1789431582.785787:0:333642:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c5712c00, open_flags 80c0, mode 81a4 opened 0 00000080:00200000:3.0:1789431582.785794:0:333642:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open|creat 00000002:00010000:3.0:1789431582.785814:0:333642:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_87bfc7da8f5002267999494b72dc20b1,[0x200000402:0x6a8:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open|creat flags 0100301 00010000:00010000:3.0:1789431582.785825:0:333642:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431582.785921:0:333642:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431582.785928:0:333642:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1789431582.785931:0:333642:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaae lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad46b expref: -99 pid: 333615 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1789431582.785958:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.785965:0:333642:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.785974:0:333642:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.785982:0:333642:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4a80 x1876350913536000/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.785991:0:333642:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333642 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.786008:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913536000:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:3.0:1789431582.786117:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.788563:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913536000:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.788585:0:333642:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788616:0:333642:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788626:0:333642:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788636:0:333642:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788647:0:333642:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788659:0:333642:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788674:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788682:0:333642:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.788694:0:333642:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x8000037, status=0 req@ffff9811d1bc4a80 x1876350913536000/t4294974012(4294974012) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1789431582.788706:0:333642:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4a80 x1876350913536000/t4294974012(4294974012) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.788717:0:333642:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788726:0:333642:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1704 (0 0) 00000080:00200000:3.0:1789431582.788732:0:333642:0:(namei.c:945:ll_intent_lock()) intent lock 3 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431582.788743:0:333642:0:(namei.c:1621:ll_create_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open|creat 00000080:00200000:3.0:1789431582.788768:0:333642:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00200000:3.0:1789431582.788774:0:333642:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a8:0x0], updating i_size 0 00000080:00200000:3.0:1789431582.788805:0:333642:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a8:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1789431582.788810:0:333642:0:(namei.c:1587:ll_create_node()) setting l_ast_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00010000:3.0:1789431582.788814:0:333642:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27ead8 00000080:00010000:3.0:1789431582.788823:0:333642:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00200008:3.0:1789431582.788864:0:333642:0:(file.c:1090:ll_file_open()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08), flags 100300 00000002:00100000:3.0:1789431582.788897:0:333642:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811d1bc4a80 x1876350913536000/t4294974012(4294974012) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1789431582.788930:0:333642:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ead8 from it ffff9811d8d9fba0 00010000:00010000:3.0:1789431582.788935:0:333642:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.788946:0:333642:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.788967:0:333642:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08), inode mode 81a4 mask 40 00000080:00200008:3.0:1789431582.788993:0:333642:0:(file.c:479:ll_file_release()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08), flags 100000 00000002:00100000:3.0:1789431582.789025:0:333642:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811d1bc4a80 x1876350913536000/t4294974012(4294974012) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1789431582.789032:0:333642:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1789431582.789035:0:333642:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7b80 x1876350913536256/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1789431582.789049:0:333642:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913536256:192.168.203.128@tcp:35:pjdfstest.0 00000100:00100000:3.0:1789431582.789102:0:333642:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.790657:0:333642:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7b80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333642:1876350913536256:192.168.203.128@tcp:35:pjdfstest.0 00000002:00100000:3.0:1789431582.790681:0:333642:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811d1bc4a80 x1876350913536000/t4294974012(4294974012) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431582.794374:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.794393:0:333645:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.794403:0:333645:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.794425:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.794493:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.794503:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.794515:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.794541:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536384:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431582.794623:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.796426:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536384:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431582.796454:0:333645:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796473:0:333645:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796487:0:333645:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796499:0:333645:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796512:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796521:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796530:0:333645:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.796542:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2100000 x1876350913536384/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.796555:0:333645:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796567:0:333645:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1703 (0 0) 00000080:00200000:3.0:1789431582.796573:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431582.796596:0:333645:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a7:0x0], updating i_size 4096 00000080:00010000:3.0:1789431582.796603:0:333645:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:3.0:1789431582.796609:0:333645:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.796612:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfbcb8 00010000:00010000:3.0:1789431582.796621:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.796631:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.798072:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.798084:0:333645:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.798088:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.798107:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.798153:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.798163:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.798174:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.798195:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536512:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.798269:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.800182:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536512:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.800198:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800208:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800214:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800221:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800229:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800236:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.800241:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800249:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800258:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800265:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.800272:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800277:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eae6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.800292:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913536512/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.800303:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.800325:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.800334:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431582.800340:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb970 00010000:00010000:3.0:1789431582.800343:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.800348:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.800416:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.800432:0:333645:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.800436:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.800451:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.800484:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800490:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.800498:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.800516:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536640:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.800578:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.802153:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536640:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.802175:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802187:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802193:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802201:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802209:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802215:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.802218:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802225:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802230:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802237:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.802242:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802249:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eaed lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.802265:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913536640/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.802279:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.802302:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.802314:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431582.802323:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb970 00010000:00010000:3.0:1789431582.802327:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.802336:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.802395:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.802405:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.802441:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.802445:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.802460:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.802464:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.802479:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.802488:0:333645:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.802492:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.802506:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.802537:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802544:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.802552:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.802570:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536768:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.802648:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.804784:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913536768:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.804806:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804819:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804826:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804833:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804887:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804894:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.804908:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804916:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804923:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804930:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.804937:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.804943:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eaf4 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.804959:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913536768/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.804974:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.804999:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.805011:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431582.805021:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb970 00010000:00010000:3.0:1789431582.805024:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.805032:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.805101:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.805111:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.805123:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.805126:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.805134:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.805147:0:333645:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c2c46000, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.805155:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.805171:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.805215:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.805222:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.805232:0:333645:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.805240:0:333645:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4380 x1876350913537024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.805250:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.805270:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537024:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.805469:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.807044:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537024:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.807063:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807075:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807082:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807090:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807099:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807105:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.807109:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807116:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807122:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807129:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.807134:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807141:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eafb lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.807159:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913537024/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.807171:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.807192:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.807200:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431582.807210:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb9a0 00010000:00010000:3.0:1789431582.807213:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.807218:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.807669:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807679:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.807694:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807699:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.807711:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807714:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.807722:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807723:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.807729:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807731:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.807735:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807737:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.807742:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807744:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.807751:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807754:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.807763:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.807828:0:333645:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c2a53500, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.807887:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.807918:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.807971:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.807979:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.807991:0:333645:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.808000:0:333645:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4380 x1876350913537280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.808010:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.808033:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537280:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.808205:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.810371:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537280:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.810393:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810407:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810414:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810421:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810429:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810439:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.810444:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810450:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810458:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810464:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.810470:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.810476:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb02 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.810494:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913537280/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.810513:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.810539:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.810564:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431582.810574:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb9a0 00010000:00010000:3.0:1789431582.810577:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.810590:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.810984:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.810993:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.811008:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811013:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.811025:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811028:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.811055:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811059:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.811077:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811080:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.811089:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811092:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.811100:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811103:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.811111:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811114:0:333645:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.811122:0:333645:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.811132:0:333645:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811d8c56480, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.811136:0:333645:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.811151:0:333645:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.811193:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.811201:0:333645:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.811212:0:333645:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.811218:0:333645:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4380 x1876350913537536/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.811227:0:333645:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.811247:0:333645:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537536:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.811380:0:333645:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.816412:0:333645:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333645:1876350913537536:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.816430:0:333645:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816439:0:333645:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816445:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816449:0:333645:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816455:0:333645:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816466:0:333645:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.816469:0:333645:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816476:0:333645:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816481:0:333645:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816485:0:333645:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.816489:0:333645:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.816493:0:333645:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb09 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333645 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.816506:0:333645:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc4380 x1876350913537536/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.816516:0:333645:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.816539:0:333645:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.816552:0:333645:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:3.0:1789431582.816561:0:333645:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1bfb9a0 00010000:00010000:3.0:1789431582.816565:0:333645:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.816574:0:333645:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431582.820989:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.821006:0:333648:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.821015:0:333648:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:3.0:1789431582.821034:0:333648:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.821051:0:333648:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431582.821080:0:333648:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:3.0:1789431582.821089:0:333648:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.821097:0:333648:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.821101:0:333648:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:3.0:1789431582.821110:0:333648:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1563cb8 00010000:00010000:3.0:1789431582.821114:0:333648:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:3.0:1789431582.821134:0:333648:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:1.0:1789431582.823401:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823414:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.823420:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823429:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.823432:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823439:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.823448:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823455:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.823456:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823462:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.823476:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823480:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823482:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.823487:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.823507:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823509:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823517:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.823518:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.823540:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823542:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823546:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.823550:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.823570:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823572:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823579:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.823579:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.823598:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823600:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.823606:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823612:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.823613:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.823618:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.823633:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823639:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.823656:0:333648:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.823660:0:333648:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.824345:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.824352:0:333647:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, flags=1 00000080:00200000:1.0:1789431582.824359:0:333647:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08), inode mode 81a4 mask 2 00000080:00200008:1.0:1789431582.824367:0:333647:0:(llite_lib.c:2428:ll_setattr_raw()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08) current size 0, valid attrs 8, mode ffff, uid -1501269716, gid -1, new size 1234567, atime 1789431582.000000000, mtime 1789431582.000000000, ctime 1789431582.000000000 00010000:00010000:1.0:1789431582.824398:0:333647:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1789431582.824420:0:333647:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6a8:0x0]: losing layout lock 00010000:00010000:1.0:1789431582.824465:0:333647:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431582.824473:0:333647:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1789431582.824476:0:333647:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118600/0x1ffb2ee1bd27ead8 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad4f0 expref: -99 pid: 333642 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1789431582.824495:0:333647:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1789431582.824508:0:333647:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdca80 x1876350913537792/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431582.824530:0:333647:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913537792:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:1.0:1789431582.824606:0:333647:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.827389:0:333647:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdca80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913537792:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:1.0:1789431582.827434:0:333647:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:1.0:1789431582.827452:0:333647:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000002:00010000:1.0:1789431582.827462:0:333647:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a8:0x0]) in obj [0x200000402:0x6a8:0x0], intent: layout flags 0 00010000:00010000:1.0:1789431582.827500:0:333647:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333647 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.827510:0:333647:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333647 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.827534:0:333647:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333647 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.827552:0:333647:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913537920:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:1.0:1789431582.827606:0:333647:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.829615:0:333647:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913537920:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431582.829639:0:333647:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.829653:0:333647:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.829667:0:333647:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9811d0bdd180 x1876350913537920/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 376/344 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1789431582.829679:0:333647:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.829690:0:333647:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 16 res 8589935618/1704 (0 0) 00000080:00200000:1.0:1789431582.829696:0:333647:0:(namei.c:945:ll_intent_lock()) intent lock 1024 on i1 [0x200000402:0x6a8:0x0] suppgids 0 0: rc 0 00000080:00010000:1.0:1789431582.829710:0:333647:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb10 00000080:00010000:1.0:1789431582.829716:0:333647:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb10 from it ffffbc42c1677ad8 00010000:00010000:1.0:1789431582.829721:0:333647:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.829731:0:333647:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431582.829751:0:333647:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431582.829761:0:333647:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1789431582.829770:0:333647:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000402:0x6a8:0x0](00000000e449eb14) being reconfigured ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1789431582.829812:0:333647:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913538048:192.168.203.128@tcp:49:pjdfstest.0 00000100:00100000:1.0:1789431582.829889:0:333647:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901780 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.833226:0:333647:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333647:1876350913538048:192.168.203.128@tcp:49:pjdfstest.0 00000080:00200000:1.0:1789431582.833273:0:333647:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a8:0x0]: layout version change: 4294967294 -> 4294967295 00010000:00010000:1.0:1789431582.833284:0:333647:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431582.833296:0:333647:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431582.833320:0:333647:0:(vvp_io.c:305:vvp_io_fini()) [0x200000402:0x6a8:0x0] ignore/verify layout 0/0, layout version -1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:1.0:1789431582.833331:0:333647:0:(llite_lib.c:2687:ll_setattr_raw()) COMPLETED file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08) current size 0, valid attrs 10000168, mode ffff, uid -1501269716, gid -1, new size 1234567, atime 1789431582.000000000, mtime 1789431582.000000000, ctime 1789431582.000000000, rc 0 00000080:00200000:2.0:1789431582.836560:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.836576:0:333650:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.836584:0:333650:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.836603:0:333650:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.836621:0:333650:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431582.836636:0:333650:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1789431582.836644:0:333650:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.836652:0:333650:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:2.0:1789431582.836655:0:333650:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:2.0:1789431582.836661:0:333650:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1c03cb8 00010000:00010000:2.0:1789431582.836665:0:333650:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1789431582.836675:0:333650:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1789431582.839031:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839043:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.839054:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839059:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839068:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839070:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839074:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839077:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839084:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839087:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839094:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839097:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839105:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839109:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839118:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839121:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839127:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839130:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431582.839239:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839242:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.839249:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839251:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839257:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839260:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839268:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839271:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839278:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839281:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839289:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839292:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839300:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839303:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839311:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839314:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839320:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839322:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.839418:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839422:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.839427:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839429:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839433:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839435:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839440:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839442:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839447:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839449:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.839453:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839455:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839460:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839461:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839467:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839468:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.839473:0:333650:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.839475:0:333650:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431582.844143:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.844165:0:333653:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.844173:0:333653:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:1.0:1789431582.844194:0:333653:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.844212:0:333653:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431582.844228:0:333653:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00000080:00200000:1.0:1789431582.844234:0:333653:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.844239:0:333653:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.844242:0:333653:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:1.0:1789431582.844247:0:333653:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c0dd3cb8 00010000:00010000:1.0:1789431582.844250:0:333653:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1789431582.844256:0:333653:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1789431582.845668:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845680:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.845699:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845704:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.845707:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845713:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.845719:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845723:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.845730:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845735:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.845736:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845742:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.845752:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845756:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.845768:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845769:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845778:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.845779:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.845784:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845788:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.845801:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845803:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845810:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.845811:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.845853:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845857:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.845859:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845874:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.845884:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845889:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.845894:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845900:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.845905:0:333653:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.845910:0:333653:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.845917:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.845922:0:333652:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.846506:0:333652:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.846520:0:333652:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=0 00000080:00200000:2.0:1789431582.846527:0:333652:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=getattr 00000002:00010000:2.0:1789431582.846547:0:333652:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_87bfc7da8f5002267999494b72dc20b1,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 01 00010000:00010000:2.0:1789431582.846602:0:333652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.846612:0:333652:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333652 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.846622:0:333652:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333652 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.846646:0:333652:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333652:1876350913538176:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.846794:0:333652:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.849016:0:333652:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333652:1876350913538176:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:0.0:1789431582.849043:0:333652:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849059:0:333652:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849083:0:333652:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849095:0:333652:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849105:0:333652:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849116:0:333652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849124:0:333652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849131:0:333652:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431582.849143:0:333652:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2101500 x1876350913538176/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431582.849163:0:333652:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849173:0:333652:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1704 (0 0) 00000080:00200000:0.0:1789431582.849179:0:333652:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1789431582.849197:0:333652:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a8:0x0], updating i_size 1234567 00000080:00200000:0.0:1789431582.849200:0:333652:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x6a8:0x0](ffff9811c4822a08): rc = 0 00000080:00200000:0.0:1789431582.849215:0:333652:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a8:0x0]: layout version change: 4294967295 -> 4294967295 00000080:00010000:0.0:1789431582.849221:0:333652:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb17 00000080:00010000:0.0:1789431582.849229:0:333652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00010000:0.0:1789431582.849242:0:333652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb17 from it ffffbc42c174bb20 00010000:00010000:0.0:1789431582.849249:0:333652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.849261:0:333652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1789431582.849291:0:333652:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1789431582.849298:0:333652:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08),name=pjdfstest_87bfc7da8f5002267999494b72dc20b1 00000002:00010000:0.0:1789431582.849317:0:333652:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a8:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:0.0:1789431582.849325:0:333652:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431582.849335:0:333652:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431582.849342:0:333652:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1789431582.849346:0:333652:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00010000:0.0:1789431582.849350:0:333652:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb17 00000080:00010000:0.0:1789431582.849354:0:333652:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb17 from it ffffbc42c174bcb8 00010000:00010000:0.0:1789431582.849357:0:333652:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789431582.849364:0:333652:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.853163:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:3.0:1789431582.853188:0:333655:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.853198:0:333655:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:3.0:1789431582.853218:0:333655:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.853249:0:333655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431582.853266:0:333655:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00000080:00200000:3.0:1789431582.853275:0:333655:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.853281:0:333655:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.853284:0:333655:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:3.0:1789431582.853289:0:333655:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1c03cb8 00010000:00010000:3.0:1789431582.853293:0:333655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 0 00010000:00010000:3.0:1789431582.853300:0:333655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:3.0:1789431582.854416:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854426:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.854439:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854443:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854451:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854455:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854462:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854465:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854473:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854475:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854483:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854486:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854494:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854496:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854504:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854507:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854514:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854523:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:3.0:1789431582.854679:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854684:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.854691:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854694:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854700:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854703:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854710:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854713:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854719:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854721:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854728:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854730:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854736:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854739:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854747:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854750:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854758:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854762:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.854909:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854915:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.854926:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854930:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854943:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854946:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854954:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854956:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854961:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854963:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.854968:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854969:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854974:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854975:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854979:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854981:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.854985:0:333655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.854987:0:333655:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431582.859157:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.859173:0:333658:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.859180:0:333658:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:1.0:1789431582.859200:0:333658:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.859233:0:333658:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:1.0:1789431582.859249:0:333658:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00000080:00200000:1.0:1789431582.859257:0:333658:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.859264:0:333658:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.859268:0:333658:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:1.0:1789431582.859275:0:333658:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1563cb8 00010000:00010000:1.0:1789431582.859279:0:333658:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 0 00010000:00010000:1.0:1789431582.859288:0:333658:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431582.860047:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860055:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.860065:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860069:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.860077:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860080:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.860091:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860094:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.860101:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860104:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.860110:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860114:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.860120:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860122:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.860129:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860131:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.860140:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860144:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:1.0:1789431582.860378:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860383:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.860392:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860394:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.860402:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860405:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.860413:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860415:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.860423:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860426:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.860433:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860435:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.860442:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860445:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.860453:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860456:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.860464:0:333658:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.860467:0:333658:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.860584:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.860590:0:333657:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, flags=1 00000080:00200000:2.0:1789431582.860595:0:333657:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08), inode mode 81a4 mask 2 00000002:00010000:2.0:1789431582.860616:0:333657:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x200000402:0x6a8:0x0], intent: getxattr flags 0 00010000:00010000:2.0:1789431582.860661:0:333657:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 3/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 333657 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.860670:0:333657:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 3/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333657 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.860680:0:333657:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 3/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT flags: 0x0 pid: 333657 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.860704:0:333657:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333657:1876350913538304:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.860777:0:333657:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9480 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.865760:0:333657:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333657:1876350913538304:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:0.0:1789431582.865788:0:333657:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.865805:0:333657:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.865817:0:333657:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431582.865860:0:333657:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x80 disposition=0x0, status=0 req@ffff9811d2102a00 x1876350913538304/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 568/1072 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431582.865884:0:333657:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.865898:0:333657:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1704 (0 0) 00000080:00200000:0.0:1789431582.865905:0:333657:0:(namei.c:945:ll_intent_lock()) intent lock 128 on i1 [0x200000402:0x6a8:0x0] suppgids 0 -1: rc 0 00000080:00010000:0.0:1789431582.865936:0:333657:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb1e 00000080:00010000:0.0:1789431582.865948:0:333657:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb1e from it ffffbc42c174bc50 00010000:00010000:0.0:1789431582.865953:0:333657:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.865965:0:333657:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1789431582.866005:0:333657:0:(llite_lib.c:2428:ll_setattr_raw()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08) current size 1234567, valid attrs 8, mode ffff, uid -1501269716, gid -1, new size 567, atime 1789431582.000000000, mtime 1789431582.000000000, ctime 1789431582.000000000 00010000:00010000:0.0:1789431582.866027:0:333657:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431582.866045:0:333657:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1789431582.866254:0:333657:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789431582.866262:0:333657:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431582.866265:0:333657:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63600/0x1ffb2ee1bd27eb17 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad5ad expref: -99 pid: 333652 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1789431582.866283:0:333657:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1789431582.866290:0:333657:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4380 x1876350913538560/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789431582.866313:0:333657:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333657:1876350913538560:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:0.0:1789431582.866381:0:333657:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.869942:0:333657:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333657:1876350913538560:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:0.0:1789431582.869991:0:333657:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a8:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789431582.870002:0:333657:0:(vvp_io.c:305:vvp_io_fini()) [0x200000402:0x6a8:0x0] ignore/verify layout 0/0, layout version -1 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:0.0:1789431582.870014:0:333657:0:(llite_lib.c:2687:ll_setattr_raw()) COMPLETED file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08) current size 1234567, valid attrs 10000168, mode ffff, uid -1501269716, gid -1, new size 567, atime 1789431582.000000000, mtime 1789431582.000000000, ctime 1789431582.000000000, rc 0 00000080:00200000:2.0:1789431582.873610:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.873627:0:333660:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.873635:0:333660:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.873655:0:333660:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.873673:0:333660:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431582.873689:0:333660:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00000080:00200000:2.0:1789431582.873698:0:333660:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.873706:0:333660:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:2.0:1789431582.873711:0:333660:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:2.0:1789431582.873719:0:333660:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1c03cb8 00010000:00010000:2.0:1789431582.873722:0:333660:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 0 00010000:00010000:2.0:1789431582.873731:0:333660:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1789431582.874932:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874943:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.874953:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874957:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.874964:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874967:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.874974:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874976:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.874984:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874987:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.874994:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.874998:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875006:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875009:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875016:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875018:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875026:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875030:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:2.0:1789431582.875192:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875197:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.875206:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875210:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875219:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875222:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875228:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875231:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875237:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875240:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875248:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875251:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875260:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875263:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875272:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875275:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875284:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875288:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:2.0:1789431582.875427:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875432:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:2.0:1789431582.875443:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875446:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875455:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875458:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875467:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875470:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875478:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875481:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.875488:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875491:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875498:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875501:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875507:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875509:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.875518:0:333660:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.875522:0:333660:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431582.880597:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.880612:0:333663:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.880620:0:333663:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:1.0:1789431582.880638:0:333663:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.880655:0:333663:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1789431582.880671:0:333663:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1789431582.880679:0:333663:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.880687:0:333663:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.880700:0:333663:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:1.0:1789431582.880707:0:333663:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1c53cb8 00010000:00010000:1.0:1789431582.880711:0:333663:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1789431582.880721:0:333663:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1789431582.881157:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881164:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.881173:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881176:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.881184:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881188:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.881194:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881197:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.881204:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881206:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.881213:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881216:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.881223:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881226:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.881233:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881236:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.881244:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881247:0:333662:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.881659:0:333662:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.881670:0:333662:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=0 00000080:00200000:3.0:1789431582.881674:0:333662:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=getattr 00000002:00010000:3.0:1789431582.881690:0:333662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_87bfc7da8f5002267999494b72dc20b1,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 01 00010000:00010000:3.0:1789431582.881735:0:333662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.881744:0:333662:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333662 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.881754:0:333662:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333662 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.881777:0:333662:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333662:1876350913538688:192.168.203.128@tcp:101:pjdfstest.0 00000080:00200000:1.0:1789431582.881804:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881809:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.881817:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881820:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.881826:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881829:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.881858:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881862:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000100:00100000:3.0:1789431582.881881:0:333662:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.881886:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881889:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.881898:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881901:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.881908:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881911:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.881919:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881922:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.881929:0:333663:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.881932:0:333663:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000100:00100000:3.0:1789431582.886418:0:333662:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333662:1876350913538688:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:3.0:1789431582.886447:0:333662:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886469:0:333662:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886478:0:333662:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886489:0:333662:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886501:0:333662:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886519:0:333662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886527:0:333662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886535:0:333662:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.886547:0:333662:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d1bc4380 x1876350913538688/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.886573:0:333662:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886583:0:333662:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1704 (0 0) 00000080:00200000:3.0:1789431582.886589:0:333662:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1789431582.886608:0:333662:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a8:0x0], updating i_size 567 00000080:00200000:3.0:1789431582.886612:0:333662:0:(namei.c:140:ll_iget()) got inode: [0x200000402:0x6a8:0x0](ffff9811c4822a08): rc = 0 00000080:00200000:3.0:1789431582.886625:0:333662:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a8:0x0]: layout version change: 4294967295 -> 4294967295 00000080:00010000:3.0:1789431582.886631:0:333662:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb25 00000080:00010000:3.0:1789431582.886640:0:333662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00010000:3.0:1789431582.886643:0:333662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb25 from it ffffbc42c174bb20 00010000:00010000:3.0:1789431582.886647:0:333662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.886656:0:333662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1789431582.886703:0:333662:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a8:0x0](ffff9811c4822a08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1789431582.886710:0:333662:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08),name=pjdfstest_87bfc7da8f5002267999494b72dc20b1 00000002:00010000:3.0:1789431582.886731:0:333662:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a8:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:3.0:1789431582.886741:0:333662:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.886752:0:333662:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.886760:0:333662:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1789431582.886764:0:333662:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00010000:3.0:1789431582.886767:0:333662:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a8:0x0](ffff9811c4822a08) for lock 0x1ffb2ee1bd27eb25 00000080:00010000:3.0:1789431582.886771:0:333662:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb25 from it ffffbc42c174bcb8 00010000:00010000:3.0:1789431582.886774:0:333662:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.886781:0:333662:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1789431582.890769:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.890785:0:333665:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.890793:0:333665:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.890813:0:333665:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.890829:0:333665:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431582.890879:0:333665:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1789431582.890888:0:333665:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.890895:0:333665:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:2.0:1789431582.890898:0:333665:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:2.0:1789431582.890905:0:333665:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c1c5bcb8 00010000:00010000:2.0:1789431582.890909:0:333665:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1789431582.890918:0:333665:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1789431582.892536:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892546:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.892557:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892561:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892568:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892571:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892578:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892581:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892589:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892592:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892599:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892602:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892610:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892613:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892622:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892625:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892631:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892635:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpcre.so.1, flags=257 00000080:00200000:0.0:1789431582.892780:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892785:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.892794:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892796:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892803:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892806:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892813:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892817:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892824:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892827:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.892857:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892861:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892886:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892890:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892898:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892902:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.892910:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.892913:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431582.893042:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893047:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.893055:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893058:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.893066:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893068:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.893076:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893079:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.893086:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893089:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.893095:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893098:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.893106:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893109:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.893116:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893118:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.893125:0:333665:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.893128:0:333665:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libpthread.so.0, flags=257 00000080:00200000:1.0:1789431582.897402:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.897415:0:333668:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.897421:0:333668:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:1.0:1789431582.897439:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.897453:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1789431582.897469:0:333668:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1789431582.897478:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1789431582.897486:0:333668:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.897490:0:333668:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eadf 00000080:00010000:1.0:1789431582.897497:0:333668:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eadf from it ffffbc42c174bcb8 00010000:00010000:1.0:1789431582.897501:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:1.0:1789431582.897510:0:333668:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1789431582.898055:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898063:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.898077:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898081:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.898089:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898091:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.898101:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898105:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.898114:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898117:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.898125:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898129:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.898137:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898140:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.898151:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898154:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.898165:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898169:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:3.0:1789431582.898774:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.898788:0:333667:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, flags=0 00000080:00200000:3.0:1789431582.898793:0:333667:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 3 00000080:00200000:3.0:1789431582.898798:0:333667:0:(namei.c:2344:ll_unlink()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88) 00010000:00010000:3.0:1789431582.898887:0:333667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:1.0:1789431582.898937:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.898943:0:333668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.898958:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.898962:0:333668:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.898972:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00010000:00010000:3.0:1789431582.899007:0:333667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.899021:0:333667:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6a8:0x0]: losing layout lock 00010000:00010000:3.0:1789431582.899030:0:333667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.899052:0:333667:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.899061:0:333667:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6a8:0x0]: losing layout lock 00010000:00010000:3.0:1789431582.899111:0:333667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.899120:0:333667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.899127:0:333667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 1/0,0 mode: --/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.899151:0:333667:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1789431582.899157:0:333667:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 4 locks packed 00010000:00010000:3.0:1789431582.899160:0:333667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb25 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad5de expref: -99 pid: 333662 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.899169:0:333667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63400/0x1ffb2ee1bd27eb1e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a8:0x0].0x0 bits 0x20/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad5bb expref: -99 pid: 333657 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.899177:0:333667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f000/0x1ffb2ee1bd27eb10 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a8:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad591 expref: -99 pid: 333647 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.899188:0:333667:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62e00/0x1ffb2ee1bd27eadf lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad505 expref: -99 pid: 333645 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000020:00100000:3.0:1789431582.899205:0:333667:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1789431582.899217:0:333667:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc4380 x1876350913538944/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 544/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431582.899239:0:333667:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333667:1876350913538944:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:3.0:1789431582.899306:0:333667:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.901558:0:333667:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4380 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333667:1876350913538944:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:3.0:1789431582.901613:0:333667:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789431582.901622:0:333667:0:(vvp_io.c:305:vvp_io_fini()) [0x200000402:0x6a8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789431582.901627:0:333667:0:(llite_lib.c:2067:ll_clear_inode()) VFS Op:inode=[0x200000402:0x6a8:0x0](ffff9811c4822a08) 00000080:00200000:1.0:1789431582.901740:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.901747:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.901767:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.901812:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.901856:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.901879:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.901898:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539072:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.901968:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.907621:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539072:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.907642:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907655:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907661:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907668:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907675:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907681:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.907696:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907707:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907713:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907720:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.907726:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907733:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb2c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.907756:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539072/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.907769:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.907881:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.907897:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.907901:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.907917:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.907958:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907965:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.907975:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.907992:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539200:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.908060:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.909961:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539200:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.909980:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.909991:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.909998:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910004:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910014:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910020:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121ec00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.910024:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910031:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910038:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910045:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.910056:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910062:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121ec00/0x1ffb2ee1bd27eb33 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.910083:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539200/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.910096:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.910163:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.910174:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.910178:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.910190:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.910223:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910231:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.910239:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.910257:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539328:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.910312:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.911948:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539328:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.911962:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.911970:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.911977:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.911984:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.911996:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912001:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.912005:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912011:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912019:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912025:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.912029:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912036:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e200/0x1ffb2ee1bd27eb3a lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.912054:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539328/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.912065:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.912114:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.912124:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.912128:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.912139:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.912165:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912177:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.912185:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.912200:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539456:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.912250:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.914357:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539456:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.914372:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914382:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914389:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914397:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914404:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914410:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121fe00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.914413:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914419:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914426:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914432:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.914436:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914442:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121fe00/0x1ffb2ee1bd27eb41 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.914462:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539456/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.914472:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.914528:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.914540:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.914544:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.914555:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.914585:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914593:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.914600:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.914616:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539584:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.914663:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.916224:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539584:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.916236:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916242:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916246:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916250:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916255:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916258:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.916260:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916264:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916267:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916271:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.916274:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916278:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f800/0x1ffb2ee1bd27eb48 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.916290:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539584/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.916297:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.916330:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.916335:0:333668:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:1.0:1789431582.916338:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:1.0:1789431582.916345:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:1.0:1789431582.916362:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916366:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.916371:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.916379:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539712:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.916416:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.918971:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539712:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.918987:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.918997:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919002:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919022:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919030:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919042:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.919046:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919052:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919058:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919065:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.919069:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919075:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27eb4f lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.919103:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539712/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.919114:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.919173:0:333668:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.919186:0:333668:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c8195980, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431582.919192:0:333668:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:1.0:1789431582.919205:0:333668:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:1.0:1789431582.919239:0:333668:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.919246:0:333668:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431582.919254:0:333668:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431582.919259:0:333668:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdd180 x1876350913539968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431582.919267:0:333668:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.919282:0:333668:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539968:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.919425:0:333668:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.922337:0:333668:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333668:1876350913539968:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.922367:0:333668:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922383:0:333668:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922391:0:333668:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922400:0:333668:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922410:0:333668:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922417:0:333668:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121e800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.922422:0:333668:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922430:0:333668:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922439:0:333668:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922447:0:333668:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.922454:0:333668:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.922462:0:333668:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eb56 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 333668 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.922507:0:333668:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913539968/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.922522:0:333668:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1789431582.927053:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.927082:0:333670:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.927092:0:333670:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.927118:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.927191:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.927205:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.927216:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.927248:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540096:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431582.927348:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9000 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.929219:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540096:192.168.203.128@tcp:101:sh.0 00010000:00010000:3.0:1789431582.929245:0:333670:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929259:0:333670:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929272:0:333670:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929284:0:333670:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929297:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929305:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929314:0:333670:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.929331:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d2102d80 x1876350913540096/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1789431582.929344:0:333670:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929357:0:333670:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1703 (0 0) 00000080:00200000:3.0:1789431582.929363:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1789431582.929388:0:333670:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a7:0x0], updating i_size 4096 00000080:00010000:3.0:1789431582.929394:0:333670:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eb5d 00000080:00010000:3.0:1789431582.929400:0:333670:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:3.0:1789431582.929404:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6bcb8 00010000:00010000:3.0:1789431582.929408:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.929416:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1789431582.930959:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.930977:0:333670:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.930983:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.931012:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.931070:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.931081:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.931095:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.931122:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540224:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.931222:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.933885:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540224:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.933944:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.933957:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.933979:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.933987:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.933996:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934002:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.934008:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934014:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934019:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934025:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.934029:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934034:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118c00/0x1ffb2ee1bd27eb64 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.934049:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913540224/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.934062:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.934089:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.934103:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1789431582.934112:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b970 00010000:00010000:3.0:1789431582.934117:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789431582.934127:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1789431582.934213:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.934229:0:333670:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.934232:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.934251:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.934283:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934288:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.934294:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.934309:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540352:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.934726:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.936160:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540352:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.936174:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936185:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936191:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936199:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936207:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936213:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.936217:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936223:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936230:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936236:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.936241:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936247:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118a00/0x1ffb2ee1bd27eb6b lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.936422:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913540352/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.936436:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.936456:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.936466:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:3.0:1789431582.936474:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b970 00010000:00010000:3.0:1789431582.936478:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1789431582.936485:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1789431582.936536:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.936544:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.936557:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.936561:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.936569:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.936572:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.936581:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.936591:0:333670:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:3.0:1789431582.936594:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:3.0:1789431582.936607:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:3.0:1789431582.936638:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936645:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.936654:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.936672:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540480:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.936732:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.938418:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540480:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.938433:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938442:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938448:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938454:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938462:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938468:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.938472:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938478:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938484:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938489:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.938494:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938500:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118e00/0x1ffb2ee1bd27eb72 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.938532:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913540480/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.938545:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.938565:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.938573:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:3.0:1789431582.938587:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b970 00010000:00010000:3.0:1789431582.938590:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1789431582.938595:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1789431582.938642:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.938650:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.938661:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.938664:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.938673:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.938683:0:333670:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811d8c57200, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.938689:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpcre.so.1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.938711:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpcre.so.1,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.938747:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.938755:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.938763:0:333670:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.938769:0:333670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7800 x1876350913540736/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.938777:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.938799:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540736:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.938981:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.941255:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540736:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.941273:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941281:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941286:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941291:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941296:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941299:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5118200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.941302:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941306:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941310:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941314:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.941318:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.941322:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5118200/0x1ffb2ee1bd27eb79 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.941332:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913540736/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.941341:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.941359:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.941367:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:3.0:1789431582.941372:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b9a0 00010000:00010000:3.0:1789431582.941374:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1789431582.941379:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1789431582.941707:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941725:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.941737:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941741:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.941748:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941751:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.941758:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941761:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.941776:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941779:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.941786:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941789:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.941796:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941799:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.941806:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941808:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.941820:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.941829:0:333670:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c2c46600, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.941995:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.942011:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.942046:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.942054:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.942063:0:333670:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.942069:0:333670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7800 x1876350913540992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.942079:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.942096:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540992:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.942232:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.944231:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913540992:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.944249:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944260:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944266:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944274:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944283:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944289:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.944293:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944299:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944305:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944311:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.944316:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944321:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119800/0x1ffb2ee1bd27eb80 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.944338:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913540992/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.944350:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.944372:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.944382:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00010000:3.0:1789431582.944391:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b9a0 00010000:00010000:3.0:1789431582.944394:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1789431582.944402:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1789431582.944715:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944725:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:3.0:1789431582.944735:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944740:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.944749:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944752:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.944759:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944762:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.944771:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944774:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:3.0:1789431582.944783:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944787:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.944796:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944799:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.944808:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944811:0:333670:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:3.0:1789431582.944820:0:333670:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:3.0:1789431582.944830:0:333670:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811d30b0480, open_flags 8000, mode 0 opened 0 00000080:00200000:3.0:1789431582.944862:0:333670:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libpthread.so.0, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:3.0:1789431582.944892:0:333670:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libpthread.so.0,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:3.0:1789431582.944937:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.944945:0:333670:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1789431582.944954:0:333670:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1789431582.944961:0:333670:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d1bc7800 x1876350913541248/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1789431582.944969:0:333670:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1789431582.944987:0:333670:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913541248:192.168.203.128@tcp:101:grep.0 00000100:00100000:3.0:1789431582.945170:0:333670:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cda6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1789431582.946747:0:333670:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job grep:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333670:1876350913541248:192.168.203.128@tcp:101:grep.0 00010000:00010000:3.0:1789431582.946763:0:333670:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946774:0:333670:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946782:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946791:0:333670:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946799:0:333670:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946805:0:333670:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c5119e00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:3.0:1789431582.946809:0:333670:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946815:0:333670:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946822:0:333670:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946829:0:333670:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1789431582.946886:0:333670:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1789431582.946893:0:333670:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c5119e00/0x1ffb2ee1bd27eb87 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333670 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1789431582.946909:0:333670:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d1bc7800 x1876350913541248/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'grep.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1789431582.946922:0:333670:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1789431582.946943:0:333670:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.946954:0:333670:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00010000:3.0:1789431582.946963:0:333670:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c1c6b9a0 00010000:00010000:3.0:1789431582.946967:0:333670:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1789431582.946976:0:333670:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431582.948966:0:333615:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:2.0:1789431582.948979:0:333615:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1789431582.948986:0:333615:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000002:00010000:2.0:1789431582.949002:0:333615:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1789431582.949015:0:333615:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431582.949026:0:333615:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1789431582.949033:0:333615:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1789431582.949039:0:333615:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:2.0:1789431582.949043:0:333615:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eb5d 00000080:00010000:2.0:1789431582.949048:0:333615:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb5d from it ffffbc42c0ddbcb8 00010000:00010000:2.0:1789431582.949051:0:333615:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1789431582.949059:0:333615:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:0.0:1789431582.950579:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950592:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:0.0:1789431582.950603:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950606:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.950613:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950616:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.950623:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950626:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.950633:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950636:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:0.0:1789431582.950643:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950647:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.950656:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950658:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.950666:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950669:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:0.0:1789431582.950677:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.950681:0:333671:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=libc.so.6, flags=257 00000080:00200000:0.0:1789431582.951353:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:0.0:1789431582.951363:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 3 00000080:00200000:0.0:1789431582.951367:0:333671:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c4d11680, open_flags 8241, mode 81b6 opened 0 00000080:00200000:0.0:1789431582.951373:0:333671:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open|creat 00000002:00010000:0.0:1789431582.951394:0:333671:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: pjdfstest_87bfc7da8f5002267999494b72dc20b1,[0x200000402:0x6a9:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open|creat flags 0101102 00010000:00010000:0.0:1789431582.951406:0:333671:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431582.951465:0:333671:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431582.951479:0:333671:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1789431582.951481:0:333671:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27eb5d lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad69b expref: -99 pid: 333670 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:0.0:1789431582.951511:0:333671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333671 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.951518:0:333671:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333671 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1789431582.951528:0:333671:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1789431582.951541:0:333671:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd5880 x1876350913541504/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1789431582.951550:0:333671:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333671 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1789431582.951570:0:333671:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5880 pname:cluuid:pid:xid:nid:opc:job dd:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333671:1876350913541504:192.168.203.128@tcp:101:dd.0 00000100:00100000:0.0:1789431582.951674:0:333671:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf9900 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.954292:0:333671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5880 pname:cluuid:pid:xid:nid:opc:job dd:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333671:1876350913541504:192.168.203.128@tcp:101:dd.0 00010000:00010000:0.0:1789431582.954313:0:333671:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/0,1 mode: --/CW res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954337:0:333671:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954346:0:333671:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/0,1 mode: --/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954370:0:333671:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954381:0:333671:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/0,1 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954393:0:333671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954398:0:333671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 5/1,1 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954402:0:333671:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1789431582.954410:0:333671:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff9811c8cd5880 x1876350913541504/t4294974017(4294974017) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/664 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1789431582.954418:0:333671:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811c8cd5880 x1876350913541504/t4294974017(4294974017) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 688/664 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1789431582.954427:0:333671:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1789431582.954438:0:333671:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954447:0:333671:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1705 (0 0) 00000080:00200000:0.0:1789431582.954454:0:333671:0:(namei.c:945:ll_intent_lock()) intent lock 3 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1789431582.954465:0:333671:0:(namei.c:1621:ll_create_it()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open|creat 00000080:00200000:0.0:1789431582.954494:0:333671:0:(llite_lib.c:3357:ll_read_inode2()) VFS Op:inode=[0x200000402:0x6a9:0x0](ffff9811c4873a88) 00000080:00200000:0.0:1789431582.954559:0:333671:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1789431582.954564:0:333671:0:(namei.c:1587:ll_create_node()) setting l_ast_data to inode [0x200000402:0x6a9:0x0](ffff9811c4873a88) 00000080:00010000:0.0:1789431582.954568:0:333671:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a9:0x0](ffff9811c4873a88) for lock 0x1ffb2ee1bd27eb8e 00000080:00010000:0.0:1789431582.954576:0:333671:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a9:0x0](ffff9811c4873a88) 00000080:00200008:0.0:1789431582.954584:0:333671:0:(file.c:1090:ll_file_open()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a9:0x0](ffff9811c4873a88), flags 101101 00000002:00100000:0.0:1789431582.954598:0:333671:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff9811c8cd5880 x1876350913541504/t4294974017(4294974017) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/664 e 0 to 0 dl 1789431598 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1789431582.954623:0:333671:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eb8e from it ffff9811d0a98d20 00010000:00010000:0.0:1789431582.954626:0:333671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.954632:0:333671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1789431582.954646:0:333671:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a9:0x0](ffff9811c4873a88), inode mode 81a4 mask 40 00000080:00200008:0.0:1789431582.954791:0:333671:0:(file.c:2650:do_file_write_iter()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a9:0x0], ppos: 0, count: 80 00000080:00200000:0.0:1789431582.954802:0:333671:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 80 bytes 00000080:00200000:0.0:1789431582.954807:0:333671:0:(file.c:2626:ll_do_tiny_write()) result: 0, original count 80 00000080:00200000:0.0:1789431582.954813:0:333671:0:(file.c:2090:ll_file_io_generic()) pjdfstest_87bfc7da8f5002267999494b72dc20b1: write ppos: 0, bytes: 80 00000080:00200000:0.0:1789431582.954818:0:333671:0:(file.c:1848:ll_io_set_mirror()) pjdfstest_87bfc7da8f5002267999494b72dc20b1: desiginated mirror: 0 00000020:00200000:0.0:1789431582.954825:0:333671:0:(cl_io.c:215:cl_io_rw_init()) header@00000000111c8ff7[0x110, 1, [0x200000402:0x6a9:0x0] hash] 00000020:00200000:0.0:1789431582.954827:0:333671:0:(cl_io.c:215:cl_io_rw_init()) io range: 2 [0, 80) 0 0 00000080:00200000:0.0:1789431582.954861:0:333671:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a9:0x0] ignore/verify layout 0/0, layout version -1 restore needed 0 00000080:00200000:0.0:1789431582.954887:0:333671:0:(file.c:2168:ll_file_io_generic()) Range lock [0, 0] 00020000:00200000:0.0:1789431582.954894:0:333671:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 0 chunk: [0, 80) 80, 80 00020000:00200000:0.0:1789431582.954905:0:333671:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789431582.954921:0:333671:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 0] 00000008:00010000:0.0:1789431582.954932:0:333671:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:0.0:1789431582.954937:0:333671:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9811c6867630, osc lock ffff9811d003d7c0, flags 0x0 00000100:00080000:0.0:1789431582.954962:0:333671:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 2460 00000100:00080000:0.0:1789431582.954974:0:333671:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431582.954984:0:333671:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 8409088, lost: 0. 00000100:00100000:0.0:1789431582.955002:0:333671:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913541632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789431582.955047:0:333671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.955057:0:333671:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x0 ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.955076:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913541632:192.168.203.128@tcp:8:dd.0 00010000:00010000:0.0:1789431582.955082:0:333671:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 3/0,1 mode: --/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->4095] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1789431582.955101:0:333671:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff9811c8cd4e00 x1876350913541760/t0(0) o101->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1789431582.955110:0:333671:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 0 seconds 00000100:00100000:0.0:1789431582.956615:0:2388:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff9811d1e45800: committing for last_committed 4294967326 gen 6 00000100:00080000:0.0:1789431582.956626:0:2388:0:(import.c:1177:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff9811d1e45800: connect to target with instance 1 00000100:00080000:0.0:1789431582.956633:0:2388:0:(import.c:1001:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff9811d1e45800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:0.0:1789431582.956643:0:2388:0:(import.c:1258:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:0.0:1789431582.956647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:0.0:1789431582.956655:0:2388:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:0.0:1789431582.956659:0:2388:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:0.0:1789431582.956670:0:2388:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 0000000096f1bb0d): req@ffff9811c8cd4e00 x1876350913541760/t0(0) o101->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 328/400 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1789431582.956691:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913541632:192.168.203.128@tcp:8:dd.0 00000100:00100000:2.0:1789431582.956755:0:333671:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.958046:0:333671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4e00 pname:cluuid:pid:xid:nid:opc:job dd:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333671:1876350913541760:192.168.203.128@tcp:101:dd.0 00010000:00010000:0.0:1789431582.958064:0:333671:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: --/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958080:0:333671:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958092:0:333671:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958105:0:333671:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:0.0:1789431582.958117:0:333671:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,2 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958128:0:333671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 5/0,2 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958135:0:333671:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1789431582.958156:0:333671:0:(vvp_io.c:1296:vvp_io_write_start()) pjdfstest_87bfc7da8f5002267999494b72dc20b1: write [0, 80) append 0 append lockpos 0 00000080:00200000:0.0:1789431582.958172:0:333671:0:(rw26.c:676:ll_write_begin()) Writing 0 of 0 to 80 bytes 00000080:00200000:0.0:1789431582.958211:0:333671:0:(rw26.c:871:ll_write_end()) pos 0, len 80, copied 80 00000080:00200000:0.0:1789431582.958222:0:333671:0:(rw26.c:929:ll_write_end()) page@00000000265bad5a[3 0000000012c21978 2 1 00000000ebe8e479 0 0 0] 00000080:00200000:0.0:1789431582.958225:0:333671:0:(rw26.c:929:ll_write_end()) vmpage @0000000044759a9e 17ffffc0001001 3:0 ffff9811c847f380 0 lru 00000080:00200000:0.0:1789431582.958232:0:333671:0:(rw26.c:929:ll_write_end()) osc-page@0000000038910b45 0: 1< 0 - - > 2< 0 0 0 0x0 0x0 | 000000002fea06a6 00000000d4b62954 > 3< 0 0 > 4< 0 0 8 8409088 - | - - - - > 5< - - - - | 0 - | 0 - -> 00000080:00200000:0.0:1789431582.958234:0:333671:0:(rw26.c:929:ll_write_end()) end page@00000000265bad5a 00000080:00200000:0.0:1789431582.958236:0:333671:0:(rw26.c:929:ll_write_end()) queued cl_page: 1. 00000080:00200000:0.0:1789431582.958239:0:333671:0:(vvp_io.c:1179:vvp_io_write_commit()) commit async pages: 1, from 0, to 80 prio 0 00000008:00010000:0.0:1789431582.958257:0:333671:0:(osc_cache.c:858:osc_extent_find()) ### extent: 00000000af1688af ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1789431582.958275:0:333671:0:(vvp_io.c:1092:vvp_set_batch_dirty()) mapping ffff9811c4873c38, count 1, dirtied 1 00000080:00200000:0.0:1789431582.958280:0:333671:0:(vvp_io.c:1202:vvp_io_write_commit()) Committed 1 pages 80 bytes, tot: 80 00000080:00200000:0.0:1789431582.958283:0:333671:0:(vvp_io.c:1362:vvp_io_write_start()) pjdfstest_87bfc7da8f5002267999494b72dc20b1: write bytes 0, result: 80 00000020:00010000:0.0:1789431582.958296:0:333671:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff9811c8678a88 (ffff9811c519e098/0) at cl_lock_release():245 00000020:00010000:0.0:1789431582.958300:0:333671:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9811c8678a88 (ffff9811c519e098/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789431582.958303:0:333671:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff9811c6867630 (ffff9811c519eab8/1) at cl_lock_cancel():142 00010000:00010000:0.0:1789431582.958308:0:333671:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1789431582.958317:0:333671:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,0 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00010000:0.0:1789431582.958331:0:333671:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9811c8678a88 (ffff9811c519e098/0) at cl_lock_fini():72 00000020:00010000:0.0:1789431582.958334:0:333671:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff9811c6867630 (ffff9811c519e098/1) at cl_lock_fini():72 00000080:00200000:0.0:1789431582.958341:0:333671:0:(vvp_io.c:537:vvp_io_advance()) advancing 80 bytes 00000080:00200000:0.0:1789431582.958348:0:333671:0:(file.c:2205:ll_file_io_generic()) Range unlock [0, 0] 00000080:00200000:0.0:1789431582.958355:0:333671:0:(vvp_io.c:305:vvp_io_fini()) [0x200000402:0x6a9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789431582.958359:0:333671:0:(file.c:2238:ll_file_io_generic()) pjdfstest_87bfc7da8f5002267999494b72dc20b1: 2 io complete with rc: 0, result: 80, restart: 0 00000080:00200000:0.0:1789431582.958366:0:333671:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000402:0x6a9:0x0] updating i_size 80 i_blocks 0 00000080:00200000:0.0:1789431582.958370:0:333671:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 80 00000080:00200008:0.0:1789431582.958400:0:333671:0:(file.c:479:ll_file_release()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a9:0x0](ffff9811c4873a88), flags 100001 00000002:00100000:0.0:1789431582.958448:0:333671:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff9811c8cd5880 x1876350913541504/t4294974017(4294974017) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/664 e 0 to 0 dl 1789431598 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1789431582.958459:0:333671:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1789431582.958464:0:333671:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811c8cd4e00 x1876350913542016/t0(0) o35->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1789431582.958481:0:333671:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4e00 pname:cluuid:pid:xid:nid:opc:job dd:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333671:1876350913542016:192.168.203.128@tcp:35:dd.0 00000100:00100000:0.0:1789431582.958530:0:333671:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c8cf93c0 going to sleep for 11 seconds 00000100:00100000:0.0:1789431582.959703:0:333671:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4e00 pname:cluuid:pid:xid:nid:opc:job dd:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333671:1876350913542016:192.168.203.128@tcp:35:dd.0 00000002:00100000:0.0:1789431582.959780:0:333671:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff9811c8cd5880 x1876350913541504/t4294974017(4294974017) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 616/664 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.962847:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200008:1.0:1789431582.962862:0:333674:0:(file.c:6296:ll_getattr_dentry()) START file pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f:[0x200000402:0x6a7:0x0](ffff9811c4823a88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1789431582.962882:0:333674:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88),name=pjdfstest_2d347c7ccea1b4cdac3a2ef4aeb91c5f 00000080:00200000:2.0:1789431582.963623:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.963634:0:333673:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:2.0:1789431582.963638:0:333673:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=glibc-hwcaps, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:2.0:1789431582.963655:0:333673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: glibc-hwcaps,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431582.963709:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.963718:0:333673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.963726:0:333673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.963747:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542144:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.963819:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000002:00010000:1.0:1789431582.964021:0:333674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a7:0x0]) in obj [0x200000402:0x6a7:0x0], intent: getattr flags 0 00010000:00010000:1.0:1789431582.964031:0:333674:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 30 res 8589935618/1703 (0 0) 00010000:00010000:1.0:1789431582.964054:0:333674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.964067:0:333674:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.964074:0:333674:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.964086:0:333674:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333674:1876350913542272:192.168.203.128@tcp:101:sh.0 00000100:00100000:1.0:1789431582.964127:0:333674:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000100:00100000:1.0:1789431582.967018:0:333674:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job sh:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333674:1876350913542272:192.168.203.128@tcp:101:sh.0 00000100:00100000:2.0:1789431582.967021:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542144:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431582.967043:0:333673:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967054:0:333674:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967056:0:333673:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967076:0:333674:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/1,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967076:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967100:0:333673:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967107:0:333673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967110:0:333674:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967123:0:333673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63200 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.967124:0:333674:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967131:0:333673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967138:0:333674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967151:0:333674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 5/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967151:0:333673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967187:0:333674:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967195:0:333673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.967199:0:333674:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff9811d0bdd180 x1876350913542272/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'sh.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1789431582.967209:0:333673:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431582.967215:0:333673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1789431582.967220:0:333674:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967223:0:333673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63200/0x1ffb2ee1bd27eb9c lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.967240:0:333674:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 4 res 8589935618/1703 (0 0) 00000002:00100000:2.0:1789431582.967247:0:333673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913542144/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 584/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1789431582.967248:0:333674:0:(namei.c:945:ll_intent_lock()) intent lock 8 on i1 [0x200000402:0x6a7:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1789431582.967263:0:333673:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.967271:0:333674:0:(llite_lib.c:3120:ll_update_inode()) inode=[0x200000402:0x6a7:0x0], updating i_size 4096 00010000:00010000:2.0:1789431582.967282:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1789431582.967291:0:333674:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) for lock 0x1ffb2ee1bd27eba3 00010000:00010000:2.0:1789431582.967296:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:1.0:1789431582.967297:0:333674:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000402:0x6a7:0x0](ffff9811c4823a88) 00000080:00010000:1.0:1789431582.967304:0:333674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c63cb8 00010000:00010000:1.0:1789431582.967310:0:333674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 4/2,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1789431582.967312:0:333673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c5b970 00010000:00010000:1.0:1789431582.967319:0:333674:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967330:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967340:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431582.967394:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.967406:0:333673:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:2.0:1789431582.967410:0:333673:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=tls, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:2.0:1789431582.967424:0:333673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: tls,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00010000:00010000:2.0:1789431582.967457:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967464:0:333673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.967472:0:333673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.967490:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542400:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.967559:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.968773:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.968783:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=glibc-hwcaps, flags=17 00000080:00200000:1.0:1789431582.968794:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:2.0:1789431582.969642:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542400:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431582.969660:0:333673:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969672:0:333673:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969679:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969698:0:333673:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969706:0:333673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969712:0:333673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62c00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431582.969716:0:333673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969723:0:333673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969730:0:333673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969736:0:333673:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431582.969742:0:333673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.969748:0:333673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62c00/0x1ffb2ee1bd27ebaa lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431582.969774:0:333673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913542400/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431582.969829:0:333673:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431582.969986:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.970000:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431582.970007:0:333673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c5b970 00010000:00010000:2.0:1789431582.970010:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.970018:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1789431582.970042:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.970077:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.970077:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.970087:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.970087:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.970104:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.970108:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:1.0:1789431582.970111:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.970121:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.970123:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00000080:00200000:2.0:1789431582.970128:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.970133:0:333673:0:(namei.c:1328:ll_lookup_nd()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), flags=17 00000080:00200000:2.0:1789431582.970136:0:333673:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=x86_64, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=0 00000002:00010000:2.0:1789431582.970149:0:333673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: x86_64,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: lookup flags 0 00000080:00200000:1.0:1789431582.970181:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00010000:00010000:2.0:1789431582.970190:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.970198:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00010000:00010000:2.0:1789431582.970199:0:333673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.970206:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=tls, flags=17 00010000:00010000:2.0:1789431582.970211:0:333673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1789431582.970220:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:2.0:1789431582.970233:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542528:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.970310:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431582.972163:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542528:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431582.972180:0:333673:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972189:0:333673:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972196:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972203:0:333673:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972210:0:333673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972216:0:333673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a63a00 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431582.972220:0:333673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972226:0:333673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972232:0:333673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972238:0:333673:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431582.972244:0:333673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972255:0:333673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63a00/0x1ffb2ee1bd27ebb1 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431582.972274:0:333673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913542528/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 576/584 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431582.972286:0:333673:0:(namei.c:945:ll_intent_lock()) intent lock 16 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1789431582.972321:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431582.972332:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:2.0:1789431582.972340:0:333673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c5b970 00010000:00010000:2.0:1789431582.972344:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789431582.972351:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1789431582.972405:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.972413:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.972426:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.972429:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:2.0:1789431582.972439:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.972450:0:333673:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811ca226900, open_flags 8000, mode 0 opened 0 00000080:00200000:2.0:1789431582.972456:0:333673:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:2.0:1789431582.972471:0:333673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431582.972505:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.972523:0:333673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1789431582.972534:0:333673:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1789431582.972540:0:333673:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913542784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1789431582.972550:0:333673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.972567:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542784:192.168.203.128@tcp:101:pjdfstest.0 00000100:00100000:2.0:1789431582.972706:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.972771:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.972795:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.972800:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.972812:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:1.0:1789431582.972815:0:333674:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=x86_64, flags=17 00000080:00200000:1.0:1789431582.972825:0:333674:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000100:00100000:2.0:1789431582.974555:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913542784:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431582.974576:0:333673:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974587:0:333673:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974594:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974605:0:333673:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974614:0:333673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974621:0:333673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811d0a62800 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:2.0:1789431582.974625:0:333673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974632:0:333673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974639:0:333673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974643:0:333673:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1789431582.974648:0:333673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.974654:0:333673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a62800/0x1ffb2ee1bd27ebb8 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431582.974671:0:333673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d2100700 x1876350913542784/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1789431582.974684:0:333673:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1789431582.974734:0:333674:0:(namei.c:1405:ll_atomic_open()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), file ffff9811c388f680, open_flags 8000, mode 0 opened 0 00000080:00200000:1.0:1789431582.974744:0:333674:0:(namei.c:1065:ll_lookup_it()) VFS Op:name=libc.so.6, dir=[0x200000402:0x6a7:0x0](ffff9811c4823a88), intent=open 00000002:00010000:1.0:1789431582.974763:0:333674:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: libc.so.6,[0x0:0x0:0x0]) in obj [0x200000402:0x6a7:0x0], intent: open flags 0100001 00010000:00010000:2.0:1789431582.974774:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431582.974787:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00010000:2.0:1789431582.974795:0:333673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c5b9a0 00010000:00010000:2.0:1789431582.974800:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1789431582.974814:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431582.974932:0:333674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.974942:0:333674:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1789431582.974951:0:333674:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1789431582.974958:0:333674:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d0bdd180 x1876350913543040/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1789431582.974967:0:333674:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1789431582.974986:0:333674:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333674:1876350913543040:192.168.203.128@tcp:101:tail.0 00000100:00100000:1.0:1789431582.975122:0:333674:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811c5901600 going to sleep for 11 seconds 00000080:00200000:2.0:1789431582.975599:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a7:0x0](ffff9811c4823a88), inode mode 41ed mask 1 00000080:00200000:2.0:1789431582.975609:0:333673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=pjdfstest_87bfc7da8f5002267999494b72dc20b1, flags=1 00000080:00200000:2.0:1789431582.975616:0:333673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000402:0x6a9:0x0](ffff9811c4873a88), inode mode 81a4 mask 2 00000080:00200008:2.0:1789431582.975624:0:333673:0:(llite_lib.c:2428:ll_setattr_raw()) START file pjdfstest_87bfc7da8f5002267999494b72dc20b1:[0x200000402:0x6a9:0x0](ffff9811c4873a88) current size 80, valid attrs 8, mode ffff, uid -1501269716, gid -1, new size 23456, atime 1789431582.000000000, mtime 1789431582.000000000, ctime 1789431582.000000000 00010000:00010000:2.0:1789431582.975645:0:333673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431582.975661:0:333673:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6a9:0x0]: losing layout lock 00010000:00010000:2.0:1789431582.975687:0:333673:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.975694:0:333673:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1789431582.975697:0:333673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9812021c3000/0x1ffb2ee1bd27eb8e lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad735 expref: -99 pid: 333671 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1789431582.975861:0:333673:0:(genops.c:2427:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff9811d1e45800: modify RPC slot 2 is allocated opc 36, max 7 00000100:00100000:2.0:1789431582.975880:0:333673:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff9811d2100700 x1876350913543296/t0(0) o36->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431582.975897:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913543296:192.168.203.128@tcp:36:pjdfstest.0 00000100:00100000:2.0:1789431582.975934:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000100:00100000:2.0:1789431582.977497:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100700 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913543296:192.168.203.128@tcp:36:pjdfstest.0 00000080:00200000:2.0:1789431582.977531:0:333673:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00010000:2.0:1789431582.977548:0:333673:0:(file.c:7065:ll_layout_intent()) ### lustre: requeue layout lock for file [0x200000402:0x6a9:0x0](ffff9811c4873a88) 00000002:00010000:2.0:1789431582.977557:0:333673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000402:0x6a9:0x0]) in obj [0x200000402:0x6a9:0x0], intent: layout flags 0 00000100:00100000:1.0:1789431582.977576:0:333674:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job tail:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333674:1876350913543040:192.168.203.128@tcp:101:tail.0 00010000:00010000:1.0:1789431582.977592:0:333674:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.977603:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977604:0:333674:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977623:0:333674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/1,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.977625:0:333673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.977644:0:333673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: --/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT flags: 0x0 pid: 333673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977645:0:333674:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977663:0:333674:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1789431582.977671:0:333673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913543424:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:1.0:1789431582.977672:0:333674:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff9811c121f600 already unused, calling callback (ffffffffc1c3e740) 00010000:00010000:1.0:1789431582.977679:0:333674:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977688:0:333674:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 4/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977696:0:333674:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 5/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977703:0:333674:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1789431582.977709:0:333674:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 3/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1789431582.977715:0:333674:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121f600/0x1ffb2ee1bd27ebbf lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a7:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 333674 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1789431582.977735:0:333674:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x7, status=0 req@ffff9811d0bdd180 x1876350913543040/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 656/608 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'tail.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1789431582.977745:0:333673:0:(client.c:2716:ptlrpc_set_wait()) set ffff9811d0ba9a80 going to sleep for 11 seconds 00000080:00200000:1.0:1789431582.977906:0:333674:0:(namei.c:945:ll_intent_lock()) intent lock 1 on i1 [0x200000402:0x6a7:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1789431582.977923:0:333674:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431582.977935:0:333674:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00010000:1.0:1789431582.977994:0:333674:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27eba3 from it ffffbc42c1c639a0 00010000:00010000:1.0:1789431582.978000:0:333674:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 3/1,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1789431582.978009:0:333674:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:2.0:1789431582.980072:0:333673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job pjdfstest:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:333673:1876350913543424:192.168.203.128@tcp:101:pjdfstest.0 00010000:00010000:2.0:1789431582.980095:0:333673:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.980107:0:333673:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 4/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1789431582.980130:0:333673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x400 disposition=0x1, status=0 req@ffff9811d2100a80 x1876350913543424/t0(0) o101->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 376/400 e 0 to 0 dl 1789431598 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1789431582.980140:0:333673:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: layout, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1789431582.980152:0:333673:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.980163:0:333673:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff9811c5215800 type 13 mode 16 res 8589935618/1705 (0 0) 00000080:00200000:2.0:1789431582.980168:0:333673:0:(namei.c:945:ll_intent_lock()) intent lock 1024 on i1 [0x200000402:0x6a9:0x0] suppgids 0 0: rc 0 00000080:00010000:2.0:1789431582.980178:0:333673:0:(llite_internal.h:1990:ll_set_lock_data()) setting l_data to inode [0x200000402:0x6a9:0x0](ffff9811c4873a88) for lock 0x1ffb2ee1bd27ebc6 00000080:00010000:2.0:1789431582.980184:0:333673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x1ffb2ee1bd27ebc6 from it ffffbc42c1c5bad8 00010000:00010000:2.0:1789431582.980187:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1789431582.980196:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980214:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980224:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 2/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1789431582.980232:0:333673:0:(file.c:6948:ll_layout_lock_set()) ### file [0x200000402:0x6a9:0x0](00000000ecae3706) being reconfigured ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1789431582.980264:0:333673:0:(file.c:6843:ll_layout_conf()) [0x200000402:0x6a9:0x0]: layout version change: 4294967294 -> 0 00010000:00010000:2.0:1789431582.980269:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 3/1,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980289:0:333673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00020000:00200000:2.0:1789431582.980311:0:333673:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789431582.980325:0:333673:0:(vvp_io.c:215:vvp_io_one_lock_index()) lock: 1 [0, 18446744073709551615] 00000008:00010000:2.0:1789431582.980337:0:333673:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 0 00000008:00010000:2.0:1789431582.980341:0:333673:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff9811d3bac830, osc lock ffff9811c85c8130, flags 0x0 00010000:00010000:2.0:1789431582.980355:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980366:0:333673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 3/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980377:0:333673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 7/0,2 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980385:0:333673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PW) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 7/0,2 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1789431582.980393:0:333673:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 6/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x810020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:2.0:1789431582.980468:0:2391:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789431582.980483:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913543552:192.168.203.128@tcp:10:pjdfstest.0 00010000:00010000:3.0:1789431583.398057:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431584.422008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431585.446661:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431586.470101:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431587.493989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789431587.622069:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913543680:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431587.622071:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913543808:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431587.622173:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913543936:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431587.622174:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913544064:192.168.203.128@tcp:400:kworker.0 00000080:00200000:1.0:1789431588.134022:0:78319:0:(vvp_io.c:1830:vvp_io_init()) [0x200000402:0x6a9:0x0] ignore/verify layout 1/0, layout version -1 restore needed 0 00020000:00200000:1.0:1789431588.134037:0:78319:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:1.0:1789431588.134072:0:78319:0:(osc_cache.c:2074:try_to_add_extent_for_io()) ### extent: 00000000af1688af ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00010000:1.0:1789431588.134091:0:78319:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 00000000af1688af ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 4/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1789431588.134166:0:78319:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0001-osc-ffff9811d1e45800 lock: 0000000038d92b6d/0x1ffb2ee1bd27eb95 lrc: 5/0,1 mode: PW/PW res: [0x2c0000401:0xc:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0x4fe1f540795ad74a expref: -99 pid: 333671 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000008:00100000:1.0:1789431588.134245:0:78319:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff9811d0bddf80 - object 0x2c0000401:12 offset 0<>4096 00000100:00100000:1.0:1789431588.134314:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913544320:192.168.203.128@tcp:16:kworker.0 00000100:00100000:0.0:1789431588.134327:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913544192:192.168.203.128@tcp:4:dd.0 00010000:00010000:0.0:1789431588.518022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431589.542185:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431590.566127:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431591.590518:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431592.614001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431592.742234:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913544576:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431592.742247:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913544448:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431592.742345:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913544832:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431592.742354:0:2391:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913544704:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431593.638157:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431594.661999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431595.686113:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431596.709989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431597.734001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789431597.862248:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545088:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431597.862353:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545344:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431597.862390:0:2392:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789431597.862397:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913544960:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431597.862430:0:2392:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545216:192.168.203.128@tcp:400:kworker.0 00010000:00010000:3.0:1789431598.757949:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789431598.885957:0:2391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431582/real 1789431582] req@ffff9811d2100a80 x1876350913543552/t0(0) o10->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:6/4 lens 440/432 e 0 to 1 dl 1789431598 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'pjdfstest.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1789431598.897185:0:2391:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789431598.897198:0:2391:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff9811d1e45800: Connection to lustre-OST0001 (at 192.168.203.128@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789431598.902850:0:2391:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1789431598.902978:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431598.902983:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431598.902987:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431598.902997:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3957 00000100:00100000:1.0:1789431598.903001:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913545600:192.168.203.128@tcp:400:kworker.0 00000100:00080000:0.0:1789431598.903017:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431598.903028:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431598.903046:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913545856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431598.903081:0:2389:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1789431598.903094:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913545472:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431598.903122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913545856:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789431598.903133:0:2389:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913545728:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431599.782005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431600.805989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431601.829996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431602.854024:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789431603.493982:0:2390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431587/real 1789431587] req@ffff9811d0bdd180 x1876350913543680/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431603.508597:0:2390:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d0bdd180 x1876350913543680/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431603.508614:0:2390:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1789431603.508618:0:2390:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.128@tcp: Connection to MGS (at 192.168.203.128@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1789431603.515917:0:2390:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.128@tcp for MGC192.168.203.128@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1789431603.515922:0:2390:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1789431603.515933:0:2390:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff9811d0bdd180 x1876350913543680/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431603.515948:0:2390:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff9811d1bc7800 x1876350913544448/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 0 dl 1789431608 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431603.515971:0:2390:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff9811d2102d80 x1876350913544960/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 0 dl 1789431613 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431603.515986:0:2390:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff9811c8cd5c00 x1876350913545472/t0(0) o400->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 224/224 e 0 to 0 dl 1789431614 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1789431603.516005:0:2390:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1789431603.516008:0:2390:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1789431603.516009:0:2390:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1789431603.516012:0:2390:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1789431603.516039:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2391:1876350913544448:192.168.203.128@tcp:400:kworker.0 00000100:00000400:1.0:1789431603.516054:0:2390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431587/real 1789431587] req@ffff9811d0bdd500 x1876350913543936/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431603.516055:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2389:1876350913545472:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431603.516081:0:2390:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d0bdd500 x1876350913543936/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431603.516089:0:2390:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:0.0:1789431603.516092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:02000400:1.0:1789431603.516094:0:2390:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff9811d1e45800: Connection to lustre-MDT0001 (at 192.168.203.128@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1789431603.516101:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1789431603.516103:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2392:1876350913544960:192.168.203.128@tcp:400:kworker.0 00000100:00080000:0.0:1789431603.516106:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431603.516117:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 208 00000100:00080000:0.0:1789431603.516132:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431603.516174:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913545984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789431603.516229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913545984:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431603.524461:0:2390:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1789431603.524476:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431603.524483:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431603.524486:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431603.524496:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 208 00000100:00100000:1.0:1789431603.524498:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2390:1876350913543680:192.168.203.128@tcp:400:kworker.0 00000100:00080000:0.0:1789431603.524515:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431603.524523:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913543936:192.168.203.128@tcp:400:kworker.0 00000008:00100000:0.0:1789431603.524527:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431603.524544:0:2390:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789431603.524556:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913546240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431603.524556:0:2390:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913546112:192.168.203.128@tcp:400:kworker.0 00000100:00080000:0.0:1789431603.524592:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431603.524597:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431603.524606:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431603.524610:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789431603.524699:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546240:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789431603.878012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789431603.989926:0:2389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431587/real 1789431587] req@ffff9811d0bdc380 x1876350913543808/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431603.989949:0:2389:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d0bdc380 x1876350913543808/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431603.989956:0:2389:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1789431603.989958:0:2389:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff9811d1e45800: Connection to lustre-MDT0000 (at 192.168.203.128@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1789431603.989963:0:2389:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1789431603.989984:0:2389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431587/real 1789431587] req@ffff9811d0bddc00 x1876350913544064/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431603.989992:0:2389:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d0bddc00 x1876350913544064/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431603 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431603.989996:0:2389:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9811d1a44800 already not connected (conn 7, was 6): CONNECTING 00000100:00100000:0.0:1789431603.990007:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913543808:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431603.990012:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913544064:192.168.203.128@tcp:400:kworker.0 00000100:00080000:2.0:1789431603.990028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431603.990035:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431603.990038:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431603.990044:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 208 00000100:00080000:2.0:1789431603.990060:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431603.990070:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431603.990107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913546368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431603.990128:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431603.990131:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1789431603.990174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546368:192.168.203.128@tcp:38:kworker.0 00000100:00000400:1.0:1789431604.581925:0:2390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431588/real 1789431588] req@ffff9811d0bddf80 x1876350913544192/t0(0) o4->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789431604 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1789431604.590407:0:2390:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9811d1a44800 already not connected (conn 7, was 6): CONNECTING 00010000:00010000:2.0:1789431604.901988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789431605.029997:0:2389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431588/real 1789431588] req@ffff9811d0bdce00 x1876350913544320/t0(0) o16->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 440/432 e 0 to 1 dl 1789431604 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:0 00000100:00080000:0.0:1789431605.030027:0:2389:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9811d1a44800 already not connected (conn 7, was 6): CONNECTING 00010000:00010000:2.0:1789431605.925978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431606.950001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431607.973994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789431608.037950:0:2391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431592/real 1789431592] req@ffff9811d1bc6300 x1876350913544704/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431608.048418:0:2391:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d1bc6300 x1876350913544704/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431608.048431:0:2391:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d2000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:3.0:1789431608.048447:0:2391:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2391:1876350913544704:192.168.203.128@tcp:400:kworker.0 00000100:00100000:1.0:1789431608.101939:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431598/real 1789431598] req@ffff9811c8cd6680 x1876350913545856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431608.101971:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913545856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431608.101992:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431608.102011:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431608.102016:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913545856:192.168.203.128@tcp:8:kworker.0 00000100:00080000:3.0:1789431608.102030:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431608.102035:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431608.102037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431608.102040:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431608.102043:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431608.102045:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431608.102047:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431608.102048:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431608.102051:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431608.102055:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3973 00000100:00080000:3.0:1789431608.102061:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789431608.102069:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431608.102076:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431608.102106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913546496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431608.102167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546496:192.168.203.128@tcp:8:kworker.0 00000100:00000400:3.0:1789431608.997938:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431592/real 1789431592] req@ffff9811d1bc4700 x1876350913544576/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431608.997969:0:2392:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d1bc4700 x1876350913544576/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431608.997978:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d1800 already not connected (conn 2, was 1): CONNECTING 00010000:00010000:2.0:1789431608.997982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789431608.997985:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431592/real 1789431592] req@ffff9811d1bc6680 x1876350913544832/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431608.997996:0:2392:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d1bc6680 x1876350913544832/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431608 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431608.998003:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9811d1a44800 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:3.0:1789431608.998017:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913544576:192.168.203.128@tcp:400:kworker.0 00000100:00100000:3.0:1789431608.998021:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913544832:192.168.203.128@tcp:400:kworker.0 00000100:00080000:3.0:1789431609.125908:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431609.125917:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431609.125920:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431609.125924:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431609.125926:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431609.125930:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431609.125933:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431609.125936:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789431610.021991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431611.045994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431612.069994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1789431613.093929:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431597/real 1789431597] req@ffff9811d2101500 x1876350913545216/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1789431613.093953:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431613.110729:0:2392:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d2101500 x1876350913545216/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431613.110742:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d2000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1789431613.110750:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431597/real 1789431597] req@ffff9811d2102a00 x1876350913545088/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431613.110771:0:2392:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d2102a00 x1876350913545088/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431613.110776:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d1800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1789431613.110779:0:2392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431597/real 1789431597] req@ffff9811d2100000 x1876350913545344/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431613.110787:0:2392:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811d2100000 x1876350913545344/t0(0) o400->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 224/224 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431613.110791:0:2392:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff9811d1a44800 already not connected (conn 8, was 6): CONNECTING 00000100:00100000:2.0:1789431613.110802:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545216:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431613.110806:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545088:192.168.203.128@tcp:400:kworker.0 00000100:00100000:2.0:1789431613.110810:0:2392:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2392:1876350913545344:192.168.203.128@tcp:400:kworker.0 00000100:00080000:2.0:1789431613.157961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.157971:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789431613.157975:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.157978:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789431613.157981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.157984:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789431613.157988:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.157992:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1789431613.221945:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431603/real 1789431603] req@ffff9811d2103800 x1876350913546368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431613.221971:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913546368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431613.221979:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431603/real 1789431603] req@ffff9811c8cd4700 x1876350913546240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431613.221987:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913546240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431613.221992:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431603/real 1789431603] req@ffff9811c8cd5c00 x1876350913545984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431613.221999:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913545984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431613 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431613.222012:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431613.222024:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431613.222029:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546368:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789431613.222036:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431613.222047:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431613.222050:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546240:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789431613.222056:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431613.222064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.222070:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431613.222072:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:2.0:1789431613.222075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789431613.222078:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913545984:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431613.222084:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 3978 00000100:00080000:2.0:1789431613.222094:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1789431613.222105:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431613.222129:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913546624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431613.222147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.222149:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431613.222150:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431613.222153:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3978 00000100:00080000:2.0:1789431613.222155:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1789431613.222157:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431613.222164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431613.222173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913546752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431613.222175:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913546624:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431613.222187:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.222189:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431613.222190:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431613.222193:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3978 00000100:00080000:2.0:1789431613.222195:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1789431613.222197:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431613.222202:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431613.222209:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913546880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431613.222215:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.222217:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789431613.222223:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431613.222225:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1789431613.222253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431613.222277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546880:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789431614.118042:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789431614.182021:0:2389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431598/real 1789431598] req@ffff9811c8cd7800 x1876350913545728/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431614.182061:0:2389:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811c8cd7800 x1876350913545728/t0(0) o400->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431614 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431614.182069:0:2389:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d2000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1789431614.182077:0:2389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431598/real 1789431598] req@ffff9811c8cd4000 x1876350913545600/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431614.182089:0:2389:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811c8cd4000 x1876350913545600/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431614 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431614.182095:0:2389:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d1800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:0.0:1789431614.182113:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913545728:192.168.203.128@tcp:400:kworker.0 00000100:00100000:0.0:1789431614.182119:0:2389:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2389:1876350913545600:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431615.142067:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431616.165996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431617.190005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431618.214015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431618.341954:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431618.341962:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431618.341966:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431618.341969:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431618.341972:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431618.341975:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431618.341979:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431618.341983:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789431619.237995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789431619.878077:0:2390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431603/real 1789431603] req@ffff9811c8cd7100 x1876350913546112/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431619.878106:0:2390:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff9811c8cd7100 x1876350913546112/t0(0) o400->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 224/224 e 0 to 1 dl 1789431619 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431619.878115:0:2390:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff9811d20d1800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1789431619.878130:0:2390:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2390:1876350913546112:192.168.203.128@tcp:400:kworker.0 00010000:00010000:2.0:1789431620.262090:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431621.287343:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431622.310862:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431623.334021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431623.461885:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.461902:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.461905:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.461910:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.461912:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.461914:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.461917:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.461919:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1789431623.461986:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431608/real 1789431608] req@ffff9811d1bc6300 x1876350913546496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431623.462011:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913546496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431623.462037:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431623.462051:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431623.462055:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546496:192.168.203.128@tcp:8:kworker.0 00000100:00080000:3.0:1789431623.462069:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.462072:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.462074:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.462076:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.462077:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.462079:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431623.462080:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431623.462082:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431623.462084:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431623.462088:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3982 00000100:00080000:3.0:1789431623.462096:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1789431623.462103:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431623.462111:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431623.462131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913547008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431623.462189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431624.358039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431625.382024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431626.406003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431627.430038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431628.453994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431628.581979:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431613/real 1789431613] req@ffff9811d2102a00 x1876350913546752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431628.582023:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913546752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431628.582037:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431613/real 1789431613] req@ffff9811d2101500 x1876350913546880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431628.582056:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789431628.582058:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913546880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431628.582077:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431628.582085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789431628.582091:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431613/real 1789431613] req@ffff9811d2100000 x1876350913546624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431628.582105:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789431628.582110:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913546624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431628.582112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582131:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431628.582138:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582143:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789431628.582157:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431628.582195:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431628.582201:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546752:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789431628.582208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431628.582213:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431628.582216:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913546880:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789431628.582221:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431628.582226:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431628.582229:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913546624:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431628.582429:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582433:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431628.582437:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431628.582447:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 3987 00000100:00080000:0.0:1789431628.582456:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789431628.582468:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431628.582503:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913547136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431628.582533:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582536:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431628.582538:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431628.582542:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3987 00000100:00080000:0.0:1789431628.582546:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789431628.582550:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431628.582574:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431628.582586:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913547264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431628.582595:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582600:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431628.582601:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431628.582604:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3987 00000100:00080000:0.0:1789431628.582608:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789431628.582613:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431628.582617:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431628.582626:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913547392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431628.582634:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431628.582637:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789431628.582692:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913547136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431628.582854:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431628.582888:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547392:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789431629.478031:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431630.501979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431631.526007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431632.550016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.1F:1789431633.061963:0:2380:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.203.128@tcp (192.168.203.128:988) the network or that node may be down. 00010000:00010000:2.0:1789431633.574006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431633.702188:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431633.702200:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431633.702203:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431633.702207:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431633.702210:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431633.702213:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431633.702216:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431633.702219:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789431634.598132:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431635.621976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431636.646043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431637.669948:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431638.694065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431638.822060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431638.822072:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431638.822075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431638.822079:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431638.822082:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431638.822085:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789431638.822088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431638.822092:0:298853:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789431639.717999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431640.742010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431641.765995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431642.789993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431643.814024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431643.941946:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431623/real 1789431623] req@ffff9811d1bc6680 x1876350913547008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431643.941974:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913547008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431643 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431643.941996:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431643.942006:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431643.942011:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547008:192.168.203.128@tcp:8:kworker.0 00000100:00080000:3.0:1789431643.942047:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431643.942054:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431643.942057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431643.942060:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431643.942063:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431643.942066:0:329502:0:(pinger.c:247:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789431643.942069:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431643.942073:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431643.942076:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431643.942084:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 3998 00000100:00080000:3.0:1789431643.942093:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1789431643.942107:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431643.942119:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431643.942153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913547520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431643.942243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431644.837988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431645.861997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431645.990073:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.128:988 00000400:00000100:0.0:1789431645.990110:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431645.990124:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431645.990187:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431643/real 1789431645] req@ffff9811d1bc4700 x1876350913547520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431645.990210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913547520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431645.990221:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431645.990225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431645.990230:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431646.886053:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431647.909982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789431648.037938:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431628/real 1789431628] req@ffff9811c8cd4000 x1876350913547136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431648.037960:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913547136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431648.037967:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431628/real 1789431628] req@ffff9811c8cd7800 x1876350913547264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431648.037974:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913547264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431648.037979:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789431628/real 1789431628] req@ffff9811c8cd5180 x1876350913547392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431648.037984:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913547392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431648.037996:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431648.038011:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431648.038014:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913547136:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431648.038018:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431648.038023:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431648.038026:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547264:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789431648.038032:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431648.038038:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431648.038041:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547392:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789431648.038044:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431648.038050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431648.038054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431648.038061:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4003 00000100:00080000:3.0:1789431648.038070:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1789431648.038080:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431648.038114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913547648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431648.038144:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431648.038147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431648.038149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431648.038153:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4003 00000100:00080000:3.0:1789431648.038157:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1789431648.038161:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431648.038171:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431648.038183:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913547776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431648.038191:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431648.038194:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431648.038197:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431648.038201:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4003 00000100:00080000:3.0:1789431648.038204:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1789431648.038208:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431648.038214:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431648.038225:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913547904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431648.038234:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913547648:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431648.038238:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431648.038245:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431648.038247:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431648.038251:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4018 00000100:00080000:3.0:1789431648.038259:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431648.038263:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431648.038268:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431648.038279:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913548032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431648.038334:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431648.038364:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431648.038386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431648.934056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431649.062170:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.128:988 00000400:00000100:0.0:1789431649.062208:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431649.062222:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431649.062273:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431649.062282:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431649.062290:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431649.062370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431648/real 1789431649] req@ffff9811d1bc5500 x1876350913547904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431649.062393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913547904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431649.062405:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431649.062418:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431649.062423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431649.062430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431648/real 1789431649] req@ffff9811d1bc7100 x1876350913548032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431649.062440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913548032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431649.062451:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431649.062455:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431649.062458:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548032:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789431649.062464:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431648/real 1789431649] req@ffff9811d1bc7800 x1876350913547648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431649.062484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913547648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431649.062491:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431649.062495:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431649.062497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913547648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431649.062502:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431648/real 1789431649] req@ffff9811d1bc5f80 x1876350913547776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431649.062509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913547776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431649.062514:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431649.062516:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431649.062518:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913547776:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789431649.957996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431650.981989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431652.005972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431653.029967:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431653.157942:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431653.157953:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431653.157956:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431653.157963:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4022 00000100:00080000:3.0:1789431653.157970:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431653.157978:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431653.158015:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913548160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431653.158050:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431653.158054:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431653.158056:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431653.158060:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4022 00000100:00080000:3.0:1789431653.158064:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431653.158069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431653.158084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431653.158098:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913548288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431653.158108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913548160:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431653.158110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431653.158118:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431653.158119:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431653.158123:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4022 00000100:00080000:3.0:1789431653.158127:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431653.158132:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431653.158137:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431653.158149:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913548416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431653.158171:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431653.158174:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431653.158175:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431653.158178:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4022 00000100:00080000:3.0:1789431653.158181:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431653.158184:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431653.158188:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431653.158197:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913548544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431653.158216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431653.158244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431653.158262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431654.053984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431655.077952:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431655.206197:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.128:988 00000400:00000100:0.0:1789431655.206241:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431655.206254:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431655.206299:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431655.206307:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431655.206314:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431655.206349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431653/real 1789431655] req@ffff9811d1bc5880 x1876350913548288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431655.206384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913548288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431655.206395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431655.206399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431655.206403:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431655.206410:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431653/real 1789431655] req@ffff9811d1bc4000 x1876350913548416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431655.206421:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913548416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431655.206428:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431655.206431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431655.206433:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431655.206439:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431653/real 1789431655] req@ffff9811d1bc5180 x1876350913548544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431655.206448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913548544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431655.206455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431655.206458:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431655.206461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548544:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789431655.206469:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431653/real 1789431655] req@ffff9811d1bc6a00 x1876350913548160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431655.206480:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913548160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431655.206486:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431655.206489:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431655.206497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913548160:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789431656.101989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431657.125979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431658.150092:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431658.278172:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431658.278180:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431658.278183:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431658.278205:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4027 00000100:00080000:1.0:1789431658.278212:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431658.278222:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431658.278251:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913548672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431658.278270:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431658.278273:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431658.278275:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431658.278279:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4027 00000100:00080000:1.0:1789431658.278282:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431658.278287:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431658.278296:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431658.278309:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913548800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431658.278319:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431658.278325:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431658.278327:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789431658.278327:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913548672:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431658.278335:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4027 00000100:00080000:1.0:1789431658.278340:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431658.278344:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431658.278360:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431658.278373:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913548928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431658.278382:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431658.278384:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431658.278385:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431658.278387:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4027 00000100:00080000:1.0:1789431658.278391:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431658.278394:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431658.278398:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789431658.278403:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431658.278410:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913549056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431658.278445:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431658/real 1789431658] req@ffff9811d0bdd500 x1876350913548672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431658.278461:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913548672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431658.278469:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431658.278471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431658.278475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913548672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431658.278506:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431658.278528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431658.278540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431658/real 1789431658] req@ffff9811d0bdd180 x1876350913548800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431658.278559:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913548800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431658.278567:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431658.278570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431658.278573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431658.278578:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431658.278597:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431658.278608:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431658/real 1789431658] req@ffff9811d0bde680 x1876350913548928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431658.278618:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913548928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431658.278625:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431658.278628:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431658.278631:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913548928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431658.278636:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431658.278655:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431658.278665:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431658/real 1789431658] req@ffff9811d0bdea00 x1876350913549056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431658.278676:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913549056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431658.278689:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431658.278692:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431658.278694:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431659.173964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431660.198049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431661.222028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431662.246007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431663.270017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431663.397961:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431663.397971:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431663.397975:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431663.397985:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4032 00000100:00080000:3.0:1789431663.397992:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431663.398002:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431663.398031:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913549184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431663.398056:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431663.398059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431663.398064:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431663.398069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4032 00000100:00080000:3.0:1789431663.398072:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431663.398078:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431663.398087:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431663.398100:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913549312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431663.398102:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913549184:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431663.398110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431663.398115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431663.398117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431663.398120:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4032 00000100:00080000:3.0:1789431663.398123:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431663.398127:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431663.398130:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431663.398138:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913549440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431663.398146:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431663.398149:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431663.398150:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431663.398153:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4032 00000100:00080000:3.0:1789431663.398156:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431663.398159:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431663.398163:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431663.398171:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913549568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431663.398206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431663.398237:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431663.398258:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431664.294099:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431664.422458:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.128:988 00000400:00000100:0.0:1789431664.422498:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431664.422512:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431664.422550:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431664.422560:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431664.422569:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431664.422632:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431663/real 1789431664] req@ffff9811d1bc7480 x1876350913549312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431664.422654:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913549312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431664.422664:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431664.422668:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431664.422672:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431664.422679:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431663/real 1789431664] req@ffff9811d1bc6a00 x1876350913549440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431664.422696:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913549440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431664.422704:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431664.422707:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431664.422710:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431664.422716:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431663/real 1789431664] req@ffff9811d1bc5180 x1876350913549568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431664.422727:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913549568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431664.422735:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431664.422738:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431664.422741:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549568:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789431664.422747:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431663/real 1789431664] req@ffff9811d1bc5c00 x1876350913549184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431664.422757:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913549184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431664.422764:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431664.422767:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431664.422770:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913549184:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789431665.318013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431666.341981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431667.366014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431668.390588:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431668.518157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431668.518167:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431668.518170:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431668.518179:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4038 00000100:00080000:1.0:1789431668.518188:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431668.518198:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431668.518227:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913549696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431668.518304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913549696:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789431668.518361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789431668.518391:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431668.518395:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431668.518397:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431668.518401:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4038 00000100:00080000:1.0:1789431668.518405:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431668.518410:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431668.518420:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431668.518433:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913549824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431668.518529:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431668/real 1789431668] req@ffff9811d0bdfb80 x1876350913549696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431668.518547:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913549696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431668.518557:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431668.518560:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431668.518564:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913549696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431668.518584:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431668.518607:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431668.518619:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431668/real 1789431668] req@ffff9811d0bdf480 x1876350913549824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431668.518630:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913549824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431668.518638:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431668.518641:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431668.518644:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549824:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789431668.518698:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431668.518702:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431668.518704:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431668.518708:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4038 00000100:00080000:1.0:1789431668.518712:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431668.518717:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431668.518722:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431668.518735:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913549952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431668.518746:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431668.518749:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431668.518751:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431668.518754:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4038 00000100:00080000:1.0:1789431668.518758:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431668.518762:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431668.518766:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431668.518776:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913550080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431668.518826:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431668.518874:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431668.518886:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431668/real 1789431668] req@ffff9811d0bded80 x1876350913549952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431668.518897:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913549952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431668.518904:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431668.518907:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431668.518910:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913549952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431668.518915:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431668.518932:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431668.518942:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431668/real 1789431668] req@ffff9811d0bdc000 x1876350913550080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431668.518952:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913550080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431668.518959:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431668.518962:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431668.518965:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431669.414021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431670.438008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431671.461988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431672.485996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431673.510011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431673.637989:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431673.637998:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431673.638002:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431673.638011:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4043 00000100:00080000:1.0:1789431673.638019:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431673.638030:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431673.638060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913550208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431673.638090:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431673.638094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431673.638095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431673.638100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4043 00000100:00080000:1.0:1789431673.638104:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431673.638109:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431673.638118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431673.638132:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913550336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431673.638136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913550208:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431673.638150:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431673.638161:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431673.638163:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431673.638168:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4043 00000100:00080000:1.0:1789431673.638172:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431673.638177:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431673.638182:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431673.638194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913550464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431673.638204:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431673.638208:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431673.638210:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431673.638214:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4043 00000100:00080000:1.0:1789431673.638218:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431673.638224:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431673.638229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550336:192.168.203.128@tcp:38:kworker.0 00000008:00100000:1.0:1789431673.638230:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431673.638274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431673.638323:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913550592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431673.638371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431674.533999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431675.558091:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431676.581958:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789431676.710187:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.203.128:988 00000400:00000100:3.0:1789431676.710242:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789431676.710258:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431676.710298:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431676.710307:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431676.710316:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431676.710391:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431673/real 1789431676] req@ffff9811d0bdc700 x1876350913550592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431676.710415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913550592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431676.710426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431676.710431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431676.710435:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550592:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789431676.710441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431673/real 1789431676] req@ffff9811d0bdd880 x1876350913550336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431676.710453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913550336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431676.710460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431676.710463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431676.710467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431676.710473:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431673/real 1789431676] req@ffff9811d0bde300 x1876350913550464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431676.710492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913550464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431676.710505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431676.710509:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431676.710512:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431676.710517:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431673/real 1789431676] req@ffff9811d0bdf100 x1876350913550208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431676.710528:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913550208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431676.710535:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431676.710538:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431676.710541:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913550208:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789431677.605985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431678.630037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431678.757943:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431678.757953:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431678.757957:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431678.757979:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4048 00000100:00080000:1.0:1789431678.757987:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431678.757998:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431678.758029:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913550720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431678.758059:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431678.758063:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431678.758065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431678.758069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4048 00000100:00080000:1.0:1789431678.758072:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431678.758077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431678.758087:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431678.758102:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913550720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431678.758103:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913550848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431678.758119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431678.758123:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431678.758125:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431678.758128:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4048 00000100:00080000:1.0:1789431678.758132:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431678.758136:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431678.758141:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431678.758153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913550976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431678.758162:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789431678.758166:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431678.758170:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431678.758172:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431678.758177:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4048 00000100:00080000:1.0:1789431678.758181:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431678.758185:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431678.758190:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431678.758199:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913551104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431678.758222:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431678/real 1789431678] req@ffff9811d0bdf800 x1876350913550720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431678.758241:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913550720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431678.758258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431678.758262:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431678.758265:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913550720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431678.758287:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431678.758310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431678.758323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431678/real 1789431678] req@ffff9811d0bdf100 x1876350913550848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431678.758335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913550848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431678.758342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431678.758346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431678.758349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431678.758355:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431678.758373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431678.758390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431678/real 1789431678] req@ffff9811d0bde300 x1876350913550976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431678.758400:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913550976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431678.758406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431678.758409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431678.758411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913550976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431678.758416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431678.758433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431678.758442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431678/real 1789431678] req@ffff9811d0bdd880 x1876350913551104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431678.758457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913551104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431678.758465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431678.758468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431678.758470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431679.653992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431680.677987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431681.702030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431682.726062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431683.750013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431683.877965:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431683.877976:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431683.877980:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431683.877991:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4053 00000100:00080000:1.0:1789431683.878000:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431683.878011:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431683.878041:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913551232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431683.878068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431683.878072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431683.878073:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431683.878077:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4053 00000100:00080000:1.0:1789431683.878081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431683.878085:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431683.878095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431683.878107:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913551360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431683.878115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913551232:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431683.878121:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431683.878127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431683.878129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431683.878133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4053 00000100:00080000:1.0:1789431683.878137:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431683.878141:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431683.878145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431683.878158:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913551488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431683.878167:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431683.878182:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431683.878185:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789431683.878190:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789431683.878193:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4053 00000100:00080000:1.0:1789431683.878198:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431683.878203:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431683.878208:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431683.878218:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913551616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431683.878253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431683/real 1789431683] req@ffff9811d0bdc700 x1876350913551232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431683.878271:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913551232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431683.878281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431683.878286:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431683.878289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913551232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431683.878312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431683.878334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431683.878345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431683/real 1789431683] req@ffff9811d0bdc000 x1876350913551360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431683.878361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913551360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431683.878369:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431683.878372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431683.878375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431683.878382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431683.878401:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431683.878413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431683/real 1789431683] req@ffff9811d0bded80 x1876350913551488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431683.878424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913551488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431683.878432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431683.878435:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431683.878439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431683.878444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431683.878463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431683.878473:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431683/real 1789431683] req@ffff9811d0bdf480 x1876350913551616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431683.878484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913551616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431683.878498:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431683.878501:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431683.878503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431684.774014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431685.798009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431686.822095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431687.845988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431688.870003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431688.997948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431688.997957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431688.997961:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431688.997971:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4058 00000100:00080000:1.0:1789431688.997980:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431688.997991:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431688.998021:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913551744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431688.998053:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431688.998057:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431688.998059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431688.998063:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4058 00000100:00080000:1.0:1789431688.998066:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431688.998071:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431688.998080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431688.998094:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913551872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431688.998104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431688.998107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431688.998109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431688.998113:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4058 00000100:00080000:1.0:1789431688.998118:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789431688.998130:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913551744:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431688.998146:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431688.998157:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431688.998173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913552000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431688.998185:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431688.998188:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431688.998190:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431688.998194:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4058 00000100:00080000:1.0:1789431688.998200:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431688.998205:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431688.998211:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789431688.998224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431688.998235:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913552128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431688.998271:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431688/real 1789431688] req@ffff9811d0bdfb80 x1876350913551744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431688.998288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913551744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431688.998298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431688.998301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431688.998304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913551744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431688.998310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431688.998331:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431688.998343:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431688/real 1789431688] req@ffff9811d0bdea00 x1876350913551872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431688.998356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913551872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431688.998364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431688.998368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431688.998371:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913551872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431688.998399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431688.998434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431688.998447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431688/real 1789431688] req@ffff9811d0bde680 x1876350913552000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431688.998459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913552000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431688.998466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431688.998471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431688.998473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431688.998480:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552128:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431688.998500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431688.998510:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431688/real 1789431688] req@ffff9811d0bdd180 x1876350913552128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431688.998520:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913552128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431688.998529:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431688.998532:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431688.998534:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431689.893954:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431690.917977:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431691.941975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431692.966066:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431693.990028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431694.117973:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431694.117981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431694.117984:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431694.117992:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4063 00000100:00080000:3.0:1789431694.118000:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431694.118008:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431694.118032:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913552256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431694.118057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431694.118060:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431694.118062:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431694.118069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4063 00000100:00080000:3.0:1789431694.118073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431694.118076:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431694.118084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431694.118095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913552384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431694.118118:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431694.118122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431694.118125:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431694.118129:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4063 00000100:00080000:3.0:1789431694.118133:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431694.118138:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431694.118143:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431694.118154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913552512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431694.118159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913552256:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431694.118173:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431694.118177:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431694.118179:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431694.118183:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4063 00000100:00080000:3.0:1789431694.118187:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431694.118193:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431694.118198:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431694.118210:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913552640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431694.118249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431694.118296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431694.118323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789431695.014040:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789431695.142158:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789431695.142205:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789431695.142221:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431695.142260:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431695.142272:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431695.142282:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789431695.142313:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431694/real 1789431695] req@ffff9811d1bc5f80 x1876350913552512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431695.142343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913552512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431695.142356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431695.142361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431695.142366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431695.142374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431694/real 1789431695] req@ffff9811d1bc7800 x1876350913552640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431695.142387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913552640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431695.142395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431695.142398:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431695.142401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552640:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789431695.142407:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431694/real 1789431695] req@ffff9811d1bc4000 x1876350913552256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431695.142424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913552256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431695.142430:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431695.142434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431695.142437:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913552256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431695.142444:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431694/real 1789431695] req@ffff9811d1bc5880 x1876350913552384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431695.142455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913552384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431695.142462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431695.142465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431695.142469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552384:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789431696.037929:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431697.061981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431698.085992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431699.110056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431699.237899:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431699.237910:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431699.237915:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431699.237924:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4068 00000100:00080000:3.0:1789431699.237933:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431699.237945:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431699.237972:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913552768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431699.237998:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431699.238002:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431699.238004:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431699.238008:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4068 00000100:00080000:3.0:1789431699.238012:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431699.238018:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431699.238027:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431699.238040:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913552896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431699.238044:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913552768:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431699.238054:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431699.238058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431699.238061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431699.238080:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4068 00000100:00080000:3.0:1789431699.238084:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431699.238089:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431699.238095:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431699.238108:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913553024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431699.238117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431699.238120:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431699.238122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431699.238126:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4068 00000800:00000100:2.0:1789431699.238130:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431699.238133:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431699.238139:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431699.238144:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431699.238156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913553152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431699.238175:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431699/real 1789431699] req@ffff9811d1bc7100 x1876350913552768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431699.238193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913552768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431699.238203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431699.238206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431699.238211:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913552768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431699.238231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431699.238252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431699.238265:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431699/real 1789431699] req@ffff9811d1bc5500 x1876350913552896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431699.238276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913552896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431699.238282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431699.238285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431699.238288:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913552896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431699.238293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431699.238309:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431699.238319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431699/real 1789431699] req@ffff9811d1bc4700 x1876350913553024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431699.238328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913553024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431699.238335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431699.238338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431699.238340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431699.238345:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553152:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431699.238375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431699.238386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431699/real 1789431699] req@ffff9811d1bc6680 x1876350913553152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431699.238396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913553152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431699.238404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431699.238406:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431699.238409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431700.133997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431701.158031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431702.182545:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431703.206017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431704.230005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431704.357979:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431704.357991:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431704.357997:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431704.358010:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4073 00000100:00080000:2.0:1789431704.358023:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431704.358049:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431704.358097:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913553280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431704.358135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431704.358139:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431704.358141:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431704.358145:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4073 00000100:00080000:2.0:1789431704.358149:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431704.358155:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431704.358167:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431704.358186:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913553408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431704.358195:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913553280:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431704.358204:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431704.358209:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431704.358211:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431704.358217:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4073 00000100:00080000:2.0:1789431704.358222:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431704.358227:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431704.358232:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431704.358246:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913553536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431704.358267:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431704.358271:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431704.358276:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789431704.358279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431704.358286:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4073 00000100:00080000:2.0:1789431704.358294:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431704.358302:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431704.358307:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431704.358325:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913553664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431704.358347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431704/real 1789431704] req@ffff9811d2100000 x1876350913553280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431704.358371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913553280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431704.358384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431704.358389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431704.358394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913553280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431704.358419:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431704.358455:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431704.358469:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431704/real 1789431704] req@ffff9811d2101500 x1876350913553408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431704.358482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913553408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431704.358489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431704.358493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431704.358497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431704.358504:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431704.358524:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431704.358535:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431704/real 1789431704] req@ffff9811d2102a00 x1876350913553536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431704.358546:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913553536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431704.358554:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431704.358557:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431704.358561:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431704.358567:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431704.358587:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431704.358598:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431704/real 1789431704] req@ffff9811d2102300 x1876350913553664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431704.358610:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913553664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431704.358619:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431704.358622:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431704.358625:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431705.254056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431706.278038:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431707.302024:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431708.326057:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431709.350064:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431709.477991:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431709.478011:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431709.478015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431709.478025:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4079 00000100:00080000:3.0:1789431709.478034:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431709.478045:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431709.478077:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913553792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431709.478103:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431709.478106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431709.478108:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431709.478113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4079 00000100:00080000:3.0:1789431709.478116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431709.478121:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431709.478129:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431709.478144:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913553920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431709.478154:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913553792:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431709.478157:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431709.478164:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431709.478165:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431709.478169:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4079 00000100:00080000:3.0:1789431709.478172:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431709.478176:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431709.478181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431709.478192:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913554048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431709.478202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431709.478206:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431709.478208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431709.478211:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4079 00000100:00080000:3.0:1789431709.478214:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431709.478220:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789431709.478224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789431709.478226:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431709.478239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913554176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431709.478263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431709/real 1789431709] req@ffff9811d1bc6300 x1876350913553792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431709.478281:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913553792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431709.478292:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431709.478296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431709.478300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913553792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431709.478322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431709.478344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431709.478356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431709/real 1789431709] req@ffff9811d1bc6680 x1876350913553920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431709.478368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913553920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431709.478375:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431709.478378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431709.478382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913553920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431709.478388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431709.478405:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431709.478415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431709/real 1789431709] req@ffff9811d1bc4700 x1876350913554048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431709.478426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913554048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431709.478439:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431709.478442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431709.478445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431709.478450:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431709.478469:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431709.478480:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431709/real 1789431709] req@ffff9811d1bc5500 x1876350913554176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431709.478490:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913554176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431709.478498:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431709.478502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431709.478504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789431710.374098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431711.398006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431712.422037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431713.446011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431714.469989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431714.597923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431714.597929:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431714.597932:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431714.597940:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4084 00000100:00080000:3.0:1789431714.597948:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431714.597955:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431714.597998:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913554304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431714.598027:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431714.598030:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431714.598031:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431714.598034:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4084 00000100:00080000:3.0:1789431714.598037:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431714.598040:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431714.598048:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431714.598057:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913554432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431714.598065:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431714.598066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913554304:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431714.598070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431714.598073:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431714.598076:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4084 00000100:00080000:3.0:1789431714.598078:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431714.598080:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431714.598083:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431714.598090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913554560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431714.598100:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431714.598102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431714.598103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431714.598105:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4084 00000100:00080000:3.0:1789431714.598108:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431714.598110:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431714.598113:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431714.598116:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431714.598122:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913554688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431714.598148:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431714/real 1789431714] req@ffff9811d1bc7100 x1876350913554304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431714.598165:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913554304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431714.598171:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431714.598174:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431714.598176:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913554304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431714.598196:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431714.598211:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431714.598220:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431714/real 1789431714] req@ffff9811d1bc5880 x1876350913554432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431714.598228:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913554432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431714.598234:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431714.598238:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431714.598241:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431714.598248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431714.598269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431714.598280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431714/real 1789431714] req@ffff9811d1bc4000 x1876350913554560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431714.598290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913554560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431714.598297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431714.598299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431714.598302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431714.598307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554688:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431714.598325:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431714.598335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431714/real 1789431714] req@ffff9811d1bc7800 x1876350913554688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431714.598345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913554688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431714.598352:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431714.598355:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431714.598358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431715.494005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431716.518056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431717.541991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431718.566078:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431719.590042:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431719.717916:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431719.717932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431719.717936:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431719.717944:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4089 00000100:00080000:3.0:1789431719.717951:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431719.717960:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431719.717988:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913554816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431719.718012:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431719.718014:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431719.718016:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431719.718019:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4089 00000100:00080000:3.0:1789431719.718022:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431719.718025:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431719.718032:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431719.718040:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913554944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431719.718048:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431719.718051:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913554816:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431719.718051:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431719.718053:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431719.718060:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4089 00000100:00080000:3.0:1789431719.718062:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431719.718064:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431719.718068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431719.718075:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913555072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431719.718081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431719.718083:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431719.718084:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431719.718087:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4089 00000100:00080000:3.0:1789431719.718089:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431719.718092:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431719.718095:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431719.718103:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431719.718104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913555200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431719.718135:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431719/real 1789431719] req@ffff9811d1bc5f80 x1876350913554816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431719.718154:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913554816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431719.718164:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431719.718168:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431719.718171:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913554816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431719.718192:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431719.718213:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431719.718225:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431719/real 1789431719] req@ffff9811d1bc5c00 x1876350913554944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431719.718237:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913554944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431719.718251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431719.718254:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431719.718257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913554944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431719.718262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431719.718280:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431719.718290:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431719/real 1789431719] req@ffff9811d1bc5180 x1876350913555072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431719.718316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913555072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431719.718325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431719.718328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431719.718331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431719.718336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431719.718361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431719.718372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431719/real 1789431719] req@ffff9811d1bc6a00 x1876350913555200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431719.718384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913555200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431719.718391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431719.718394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431719.718397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431720.614004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431721.638016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431722.662014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431723.686014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431724.709975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431724.837927:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431724.837936:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431724.837939:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431724.837946:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4094 00000100:00080000:3.0:1789431724.837954:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431724.837963:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431724.837990:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913555328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431724.838010:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431724.838012:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431724.838013:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431724.838016:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4094 00000100:00080000:3.0:1789431724.838018:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431724.838020:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431724.838026:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431724.838035:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913555456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431724.838041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431724.838043:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431724.838044:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431724.838046:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4094 00000100:00080000:3.0:1789431724.838048:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431724.838051:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431724.838055:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431724.838058:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913555328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431724.838066:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913555584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431724.838078:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431724.838082:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431724.838083:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431724.838087:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4094 00000100:00080000:3.0:1789431724.838092:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431724.838097:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431724.838102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431724.838118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431724.838127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913555712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431724.838156:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431724/real 1789431724] req@ffff9811d1bc7480 x1876350913555328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431724.838173:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913555328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431724.838183:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431724.838187:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431724.838191:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913555328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431724.838198:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431724.838218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431724.838230:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431724/real 1789431724] req@ffff9811d1bc6a00 x1876350913555456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431724.838240:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913555456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431724.838248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431724.838250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431724.838253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431724.838272:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431724.838289:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431724.838300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431724/real 1789431724] req@ffff9811d1bc5180 x1876350913555584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431724.838310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913555584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431724.838317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431724.838320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431724.838323:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431724.838328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431724.838344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431724.838354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431724/real 1789431724] req@ffff9811d1bc5c00 x1876350913555712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431724.838365:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913555712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431724.838374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431724.838377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431724.838380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789431725.733993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431726.758016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431727.782026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431728.805996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431729.829984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431729.957941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431729.957951:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431729.957955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431729.957965:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4099 00000100:00080000:3.0:1789431729.957973:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431729.957983:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431729.958011:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913555840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431729.958033:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431729.958036:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431729.958038:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431729.958042:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4099 00000100:00080000:3.0:1789431729.958045:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431729.958050:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431729.958059:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431729.958068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913555968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431729.958076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431729.958079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431729.958081:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789431729.958083:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913555840:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431729.958085:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4099 00000100:00080000:3.0:1789431729.958089:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431729.958094:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431729.958097:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431729.958105:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913556096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431729.958111:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431729.958113:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431729.958114:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431729.958116:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4099 00000100:00080000:3.0:1789431729.958118:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431729.958121:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431729.958124:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431729.958131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913556224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431729.958177:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431729.958206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431729.958226:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431730.854011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431731.878001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431732.901992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789431733.030225:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789431733.030267:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789431733.030282:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431733.030323:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431733.030333:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431733.030341:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789431733.030409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431729/real 1789431733] req@ffff9811d1bc7800 x1876350913555968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431733.030434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913555968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431733.030445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431733.030449:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431733.030452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913555968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431733.030459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431729/real 1789431733] req@ffff9811d1bc4000 x1876350913556096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431733.030471:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913556096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431733.030484:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431733.030488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431733.030491:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431733.030497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431729/real 1789431733] req@ffff9811d1bc5880 x1876350913556224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431733.030507:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913556224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431733.030516:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431733.030519:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431733.030522:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556224:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789431733.030530:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431729/real 1789431733] req@ffff9811d1bc5f80 x1876350913555840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431733.030542:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913555840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431733.030550:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431733.030553:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431733.030556:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913555840:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789431733.925989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431734.949964:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431735.077977:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431735.077990:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431735.077995:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431735.078010:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4104 00000100:00080000:3.0:1789431735.078022:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431735.078036:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431735.078080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913556352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431735.078121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431735.078125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431735.078127:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431735.078132:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4104 00000100:00080000:3.0:1789431735.078136:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431735.078141:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431735.078154:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431735.078167:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913556480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431735.078176:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431735.078179:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431735.078181:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431735.078185:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4104 00000100:00080000:3.0:1789431735.078188:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431735.078193:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431735.078198:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431735.078208:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913556608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431735.078217:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431735.078220:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431735.078222:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431735.078226:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4104 00000100:00080000:3.0:1789431735.078229:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431735.078233:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431735.078239:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431735.078249:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913556736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431735.078373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913556352:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789431735.078446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431735.078488:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431735/real 1789431735] req@ffff9811d1bc7100 x1876350913556352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431735.078506:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913556352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431735.078517:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431735.078521:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431735.078526:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913556352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431735.078532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431735.078551:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431735.078561:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431735/real 1789431735] req@ffff9811d1bc5500 x1876350913556480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431735.078572:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913556480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431735.078580:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431735.078584:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431735.078587:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431735.078592:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431735.078612:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431735.078621:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431735/real 1789431735] req@ffff9811d1bc4700 x1876350913556608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431735.078630:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913556608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431735.078636:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431735.078639:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431735.078642:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431735.078646:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431735.078663:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431735.078673:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431735/real 1789431735] req@ffff9811d1bc6680 x1876350913556736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431735.078684:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913556736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431735.078693:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431735.078696:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431735.078700:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431735.974098:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431736.997988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431738.022008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431739.046004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431740.069975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431740.197890:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431740.197897:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431740.197900:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431740.197908:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4109 00000100:00080000:3.0:1789431740.197915:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431740.197923:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431740.197955:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913556864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431740.197979:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431740.197981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431740.197982:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431740.197985:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4109 00000100:00080000:3.0:1789431740.197988:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431740.197998:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431740.198005:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431740.198016:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913556864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431740.198018:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913556992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431740.198029:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431740.198032:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431740.198033:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431740.198035:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4109 00000100:00080000:3.0:1789431740.198038:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431740.198041:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431740.198046:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431740.198056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913557120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431740.198066:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431740.198070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431740.198072:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431740.198077:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4109 00000800:00000100:1.0:1789431740.198078:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431740.198084:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431740.198091:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431740.198097:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431740.198109:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913557248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431740.198122:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431740/real 1789431740] req@ffff9811d1bc6300 x1876350913556864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431740.198136:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913556864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431740.198145:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431740.198149:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431740.198151:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913556864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431740.198167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431740.198182:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431740.198189:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431740/real 1789431740] req@ffff9811d1bc6680 x1876350913556992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431740.198196:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913556992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431740.198201:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431740.198202:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431740.198204:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913556992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431740.198207:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431740.198215:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431740.198221:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431740/real 1789431740] req@ffff9811d1bc4700 x1876350913557120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431740.198227:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913557120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431740.198235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431740.198237:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431740.198239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431740.198242:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557248:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431740.198252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431740.198257:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431740/real 1789431740] req@ffff9811d1bc5500 x1876350913557248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431740.198263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913557248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431740.198268:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431740.198269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431740.198271:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789431741.093980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431742.117971:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431743.141933:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431744.165933:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431745.189998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431745.317929:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431745.317938:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431745.317943:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431745.317952:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4114 00000100:00080000:3.0:1789431745.317961:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431745.317972:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431745.318005:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913557376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431745.318028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431745.318031:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431745.318034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431745.318039:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4114 00000100:00080000:3.0:1789431745.318043:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431745.318050:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431745.318056:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913557376:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789431745.318059:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431745.318074:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913557504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431745.318084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431745.318088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431745.318089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431745.318093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4114 00000100:00080000:3.0:1789431745.318099:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431745.318103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789431745.318107:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789431745.318109:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431745.318121:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913557632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431745.318131:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431745.318135:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431745.318136:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789431745.318138:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431745/real 1789431745] req@ffff9811d1bc7100 x1876350913557376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431745.318143:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4114 00000100:00080000:3.0:1789431745.318151:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431745.318157:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431745.318160:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913557376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789431745.318164:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789431745.318172:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431745.318176:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431745.318181:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913557376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431745.318181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913557760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431745.318202:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431745.318218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431745.318225:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431745/real 1789431745] req@ffff9811d1bc5f80 x1876350913557504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431745.318232:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913557504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431745.318236:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431745.318238:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431745.318240:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431745.318243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431745.318252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431745.318260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431745/real 1789431745] req@ffff9811d1bc5880 x1876350913557632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431745.318270:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913557632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431745.318278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431745.318281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431745.318284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431745.318289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431745.318308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431745.318318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431745/real 1789431745] req@ffff9811d1bc4000 x1876350913557760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431745.318328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913557760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431745.318336:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431745.318338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431745.318341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913557760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431746.213985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431747.237969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431748.262002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431749.285978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789431750.309977:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431750.437959:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431750.437969:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431750.437974:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431750.437985:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4120 00000100:00080000:3.0:1789431750.437996:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431750.438008:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431750.438046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913557888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431750.438071:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431750.438074:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431750.438076:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431750.438080:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4120 00000100:00080000:3.0:1789431750.438083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431750.438087:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431750.438096:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431750.438114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913558016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431750.438115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913557888:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431750.438132:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431750.438135:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431750.438136:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431750.438140:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4120 00000100:00080000:3.0:1789431750.438143:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431750.438148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431750.438152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431750.438163:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913558144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431750.438187:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431750.438224:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431750/real 1789431750] req@ffff9811d1bc7800 x1876350913557888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431750.438242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913557888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431750.438251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431750.438254:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431750.438258:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913557888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431750.438279:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431750.438302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431750.438315:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431750.438318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431750/real 1789431750] req@ffff9811d1bc5c00 x1876350913558016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431750.438320:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431750.438324:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431750.438332:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4120 00000100:00080000:3.0:1789431750.438337:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431750.438340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913558016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431750.438345:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431750.438355:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789431750.438357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431750.438363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431750.438366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789431750.438373:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913558272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431750.438381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431750.438403:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431750.438414:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431750/real 1789431750] req@ffff9811d1bc5180 x1876350913558144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431750.438425:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913558144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431750.438433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431750.438437:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431750.438440:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431750.438451:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431750.438471:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431750.438482:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431750/real 1789431750] req@ffff9811d1bc6a00 x1876350913558272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431750.438492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913558272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431750.438504:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431750.438506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431750.438509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431751.333998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431752.358054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431753.382052:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431754.406010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431755.430049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431755.557949:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431755.557959:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431755.557963:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431755.557974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4125 00000100:00080000:0.0:1789431755.557983:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431755.557994:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431755.558039:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913558400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431755.558069:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431755.558073:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431755.558075:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431755.558079:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4125 00000100:00080000:0.0:1789431755.558083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431755.558089:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431755.558100:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431755.558111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913558400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789431755.558116:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913558528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431755.558128:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431755.558132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431755.558134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431755.558137:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4125 00000100:00080000:0.0:1789431755.558141:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431755.558146:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431755.558150:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431755.558159:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913558656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431755.558166:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431755.558169:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431755.558170:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431755.558173:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4125 00000100:00080000:0.0:1789431755.558176:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431755.558178:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431755.558181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431755.558186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789431755.558190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913558784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431755.558232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431755/real 1789431755] req@ffff9811c8cd7480 x1876350913558400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431755.558252:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913558400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431755.558267:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431755.558270:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431755.558273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913558400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431755.558293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431755.558310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431755.558319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431755/real 1789431755] req@ffff9811c8cd6300 x1876350913558528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431755.558326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913558528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431755.558333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431755.558336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431755.558339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431755.558345:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431755.558364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431755.558374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431755/real 1789431755] req@ffff9811c8cd7b80 x1876350913558656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431755.558387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913558656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431755.558392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431755.558394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431755.558396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431755.558400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558784:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431755.558415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431755.558421:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431755/real 1789431755] req@ffff9811c8cd6a00 x1876350913558784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431755.558429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913558784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431755.558436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431755.558439:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431755.558442:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913558784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431756.453989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431757.478090:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431758.502061:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431759.526080:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431760.549974:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431760.677961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431760.677968:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431760.677972:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431760.677984:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4130 00000100:00080000:1.0:1789431760.677993:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431760.678005:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431760.678039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913558912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431760.678065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431760.678069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431760.678070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431760.678075:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4130 00000100:00080000:1.0:1789431760.678078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431760.678082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431760.678090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431760.678099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913558912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431760.678104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913559040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431760.678115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431760.678119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431760.678121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431760.678124:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4130 00000100:00080000:1.0:1789431760.678127:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431760.678132:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431760.678137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431760.678148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913559168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431760.678159:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789431760.678171:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431760.678175:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431760.678177:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431760.678181:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4130 00000100:00080000:1.0:1789431760.678196:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431760.678202:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431760.678207:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431760.678212:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431760/real 1789431760] req@ffff9811d0bdd500 x1876350913558912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431760.678219:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913559296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431760.678239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913558912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431760.678251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431760.678255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431760.678259:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913558912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431760.678276:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431760.678301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431760.678314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431760/real 1789431760] req@ffff9811d0bddc00 x1876350913559040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431760.678325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913559040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431760.678333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431760.678336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431760.678339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431760.678345:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431760.678359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431760.678366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431760/real 1789431760] req@ffff9811d0bdc380 x1876350913559168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431760.678372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913559168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431760.678377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431760.678379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431760.678381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431760.678384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431760.678393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431760.678398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431760/real 1789431760] req@ffff9811d0bdd180 x1876350913559296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431760.678408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913559296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431760.678413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431760.678415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431760.678416:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431761.574058:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431762.598072:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431763.621995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431764.646001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431765.670006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431765.797933:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431765.797943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431765.797948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431765.797957:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4135 00000100:00080000:0.0:1789431765.797965:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431765.797976:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431765.798003:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913559424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431765.798022:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431765.798024:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431765.798026:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431765.798030:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4135 00000100:00080000:0.0:1789431765.798032:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431765.798035:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431765.798044:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431765.798056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913559552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431765.798066:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431765.798071:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431765.798073:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431765.798077:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4135 00000100:00100000:2.0:1789431765.798077:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913559424:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431765.798083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431765.798089:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431765.798094:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431765.798107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913559680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431765.798115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431765.798118:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431765.798119:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431765.798122:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4135 00000100:00080000:0.0:1789431765.798126:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431765.798130:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:2.0:1789431765.798139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789431765.798141:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431765.798160:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913559808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431765.798177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431765/real 1789431765] req@ffff9811c8cd4a80 x1876350913559424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431765.798195:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913559424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431765.798204:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431765.798208:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431765.798212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913559424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431765.798231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431765.798251:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431765.798262:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431765/real 1789431765] req@ffff9811c8cd5f80 x1876350913559552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431765.798279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913559552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431765.798286:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431765.798290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431765.798293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431765.798298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431765.798316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431765.798328:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431765/real 1789431765] req@ffff9811c8cd6d80 x1876350913559680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431765.798340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913559680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431765.798347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431765.798350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431765.798353:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431765.798359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559808:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431765.798375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431765.798383:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431765/real 1789431765] req@ffff9811c8cd5500 x1876350913559808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431765.798393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913559808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431765.798407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431765.798410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431765.798413:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913559808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431766.694000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431767.718012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431768.741984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431769.765991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431770.789984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431770.917940:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431770.917948:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431770.917951:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431770.917958:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4140 00000100:00080000:2.0:1789431770.917964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431770.917973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431770.917999:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913559936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431770.918019:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431770.918021:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431770.918023:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431770.918025:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4140 00000100:00080000:2.0:1789431770.918027:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431770.918029:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431770.918036:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431770.918044:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913560064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431770.918055:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913559936:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431770.918058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431770.918062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431770.918063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431770.918065:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4140 00000100:00080000:2.0:1789431770.918067:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431770.918069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431770.918073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431770.918080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913560192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431770.918086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431770.918088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431770.918089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431770.918091:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4140 00000100:00080000:2.0:1789431770.918094:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431770.918096:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431770.918107:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431770.918115:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431770.918120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913560320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431770.918148:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431770/real 1789431770] req@ffff9811d2103b80 x1876350913559936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431770.918166:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913559936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431770.918176:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431770.918180:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431770.918183:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913559936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431770.918203:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431770.918225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431770.918238:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431770/real 1789431770] req@ffff9811d2101c00 x1876350913560064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431770.918250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913560064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431770.918258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431770.918261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431770.918265:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431770.918271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431770.918290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431770.918301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431770/real 1789431770] req@ffff9811d2101180 x1876350913560192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431770.918312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913560192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431770.918319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431770.918323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431770.918326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431770.918331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431770.918350:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431770.918360:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431770/real 1789431770] req@ffff9811d2100e00 x1876350913560320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431770.918372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913560320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431770.918386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431770.918389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431770.918392:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431771.813987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431772.837972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431773.861956:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431774.886018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431775.909932:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431776.037916:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431776.037928:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431776.037933:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431776.037939:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4145 00000100:00080000:0.0:1789431776.037945:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431776.037953:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431776.037981:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913560448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431776.037999:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431776.038002:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431776.038003:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431776.038006:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4145 00000100:00080000:0.0:1789431776.038007:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431776.038010:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431776.038016:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431776.038023:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913560576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431776.038030:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431776.038032:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431776.038034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431776.038036:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4145 00000100:00080000:0.0:1789431776.038038:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431776.038043:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431776.038045:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913560448:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789431776.038046:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431776.038055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913560704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431776.038061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431776.038063:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431776.038065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431776.038067:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4145 00000100:00080000:0.0:1789431776.038069:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431776.038071:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431776.038074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431776.038081:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913560832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431776.038102:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431776.038132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431776/real 1789431776] req@ffff9811c8cd5500 x1876350913560448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431776.038144:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913560448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431776.038150:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431776.038153:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431776.038155:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913560448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431776.038160:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431776.038174:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431776.038181:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431776/real 1789431776] req@ffff9811c8cd6d80 x1876350913560576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431776.038187:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913560576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431776.038192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431776.038194:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431776.038195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431776.038211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431776.038220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431776.038225:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431776/real 1789431776] req@ffff9811c8cd5f80 x1876350913560704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431776.038231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913560704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431776.038236:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431776.038237:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431776.038239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431776.038243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431776.038259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431776.038268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431776/real 1789431776] req@ffff9811c8cd4a80 x1876350913560832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431776.038279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913560832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431776.038288:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431776.038291:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431776.038294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913560832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431776.934010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431777.957976:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431778.981988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431780.005980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431781.029970:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431781.157953:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431781.157963:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431781.157968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431781.157979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4150 00000100:00080000:1.0:1789431781.157987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431781.157999:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431781.158034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913560960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431781.158062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431781.158065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431781.158067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431781.158071:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4150 00000100:00080000:1.0:1789431781.158075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431781.158079:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431781.158088:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431781.158100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913561088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431781.158109:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913560960:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431781.158112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431781.158118:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431781.158120:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431781.158123:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4150 00000100:00080000:1.0:1789431781.158125:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431781.158127:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431781.158131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431781.158138:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913561216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431781.158149:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431781.158151:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431781.158152:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431781.158154:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4150 00000100:00080000:1.0:1789431781.158156:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431781.158158:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431781.158161:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431781.158168:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913561344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431781.158175:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431781.158215:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431781/real 1789431781] req@ffff9811d0bde680 x1876350913560960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431781.158233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913560960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431781.158243:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431781.158248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431781.158251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913560960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431781.158273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431781.158296:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431781.158309:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431781/real 1789431781] req@ffff9811d0bdea00 x1876350913561088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431781.158321:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913561088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431781.158329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431781.158331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431781.158334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431781.158340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431781.158369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431781.158382:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431781/real 1789431781] req@ffff9811d0bdfb80 x1876350913561216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431781.158395:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913561216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431781.158403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431781.158405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431781.158409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431781.158414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561344:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431781.158434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431781.158446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431781/real 1789431781] req@ffff9811d0bdf480 x1876350913561344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431781.158457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913561344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431781.158465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431781.158473:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431781.158476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431782.054001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431783.077975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431784.102003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431785.126048:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431786.149978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431786.277952:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431786.277962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431786.277966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431786.277976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4155 00000100:00080000:0.0:1789431786.277986:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431786.277999:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431786.278051:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913561472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431786.278079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431786.278082:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431786.278083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431786.278087:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4155 00000100:00080000:0.0:1789431786.278089:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431786.278093:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431786.278103:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431786.278111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913561600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431786.278122:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431786.278125:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431786.278127:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431786.278129:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4155 00000100:00080000:0.0:1789431786.278132:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431786.278134:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431786.278137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431786.278143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913561728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431786.278151:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431786.278154:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431786.278155:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431786.278159:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4155 00000100:00080000:0.0:1789431786.278163:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431786.278167:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431786.278168:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913561472:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789431786.278171:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431786.278184:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913561856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431786.278248:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431786.278291:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431786/real 1789431786] req@ffff9811c8cd6a00 x1876350913561472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431786.278312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913561472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431786.278322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431786.278326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431786.278329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913561472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431786.278336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431786.278355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431786.278366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431786/real 1789431786] req@ffff9811c8cd7b80 x1876350913561600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431786.278376:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913561600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431786.278383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431786.278386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431786.278389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431786.278410:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431786.278426:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431786.278436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431786/real 1789431786] req@ffff9811c8cd6300 x1876350913561728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431786.278446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913561728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431786.278478:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431786.278482:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431786.278485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431786.278490:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431786.278508:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431786.278519:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431786/real 1789431786] req@ffff9811c8cd7480 x1876350913561856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431786.278529:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913561856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431786.278537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431786.278540:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431786.278542:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913561856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431787.174031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431788.197986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431789.221990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431790.245981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431791.269973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431791.397914:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431791.397921:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431791.397924:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431791.397932:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4160 00000100:00080000:0.0:1789431791.397939:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431791.397946:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431791.397976:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913561984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431791.398001:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431791.398004:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431791.398006:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431791.398010:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4160 00000100:00080000:0.0:1789431791.398014:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431791.398019:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431791.398027:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431791.398037:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913561984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789431791.398038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913562112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431791.398050:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431791.398052:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431791.398054:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431791.398056:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4160 00000100:00080000:0.0:1789431791.398058:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431791.398061:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431791.398064:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431791.398072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913562240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431791.398078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431791.398081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431791.398082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431791.398086:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4160 00000800:00000100:2.0:1789431791.398086:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789431791.398091:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431791.398094:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431791.398098:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431791.398105:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913562368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431791.398119:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431791/real 1789431791] req@ffff9811c8cd5180 x1876350913561984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431791.398131:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913561984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431791.398145:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431791.398149:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431791.398152:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913561984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431791.398171:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431791.398192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431791.398205:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431791/real 1789431791] req@ffff9811c8cd7800 x1876350913562112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431791.398217:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913562112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431791.398225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431791.398228:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431791.398230:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431791.398234:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431791.398248:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431791.398255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431791/real 1789431791] req@ffff9811c8cd4000 x1876350913562240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431791.398261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913562240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431791.398266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431791.398268:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431791.398269:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431791.398273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431791.398282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431791.398288:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431791/real 1789431791] req@ffff9811c8cd7100 x1876350913562368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431791.398298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913562368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431791.398305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431791.398308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431791.398311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431792.293976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431793.317961:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431794.341968:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431795.365952:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431796.390011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431796.517915:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431796.517924:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431796.517928:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431796.517936:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4166 00000100:00080000:2.0:1789431796.517944:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431796.517955:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431796.517985:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913562496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431796.518008:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431796.518010:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431796.518013:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431796.518017:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4166 00000100:00080000:2.0:1789431796.518020:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431796.518026:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431796.518035:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431796.518043:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913562496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431796.518050:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913562624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431796.518062:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431796.518065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431796.518067:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431796.518071:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4166 00000100:00080000:2.0:1789431796.518075:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431796.518079:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431796.518084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431796.518095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913562752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431796.518102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431796.518104:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431796.518105:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431796.518108:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4166 00000100:00080000:2.0:1789431796.518111:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431796.518115:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431796.518119:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431796.518129:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913562880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431796.518130:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431796.518157:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431796.518174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431797.413963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431798.438046:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431799.462057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431799.590087:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789431799.590124:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431799.590138:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431799.590174:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431799.590183:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431799.590190:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789431799.590226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431796/real 1789431799] req@ffff9811d2101180 x1876350913562880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431799.590251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913562880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431799.590262:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431799.590267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431799.590271:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562880:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789431799.590278:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431796/real 1789431799] req@ffff9811d2101880 x1876350913562624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431799.590290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913562624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431799.590297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431799.590301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431799.590304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431799.590311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431796/real 1789431799] req@ffff9811d2100e00 x1876350913562752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431799.590322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913562752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431799.590327:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431799.590330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431799.590334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913562752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431799.590340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431796/real 1789431799] req@ffff9811d2101f80 x1876350913562496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431799.590350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913562496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431799.590358:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431799.590361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431799.590364:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913562496:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789431800.485988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431801.510007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431801.637941:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431801.637948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431801.637952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431801.637959:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4171 00000100:00080000:1.0:1789431801.637965:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431801.637973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431801.637997:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913563008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431801.638019:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431801.638022:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431801.638024:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431801.638028:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4171 00000100:00080000:1.0:1789431801.638031:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431801.638036:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431801.638045:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431801.638058:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913563136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431801.638066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913563008:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431801.638070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431801.638076:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431801.638078:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431801.638082:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4171 00000100:00080000:1.0:1789431801.638085:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431801.638089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431801.638095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431801.638106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913563264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431801.638118:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431801.638121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431801.638122:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431801.638125:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4171 00000100:00080000:1.0:1789431801.638129:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431801.638133:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431801.638137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789431801.638143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431801.638151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913563392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431801.638184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431801/real 1789431801] req@ffff9811d0bded80 x1876350913563008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431801.638201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913563008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431801.638211:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431801.638214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431801.638218:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913563008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431801.638238:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431801.638261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431801.638274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431801/real 1789431801] req@ffff9811d0bdc000 x1876350913563136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431801.638285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913563136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431801.638292:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431801.638296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431801.638298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431801.638304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431801.638324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431801.638335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431801/real 1789431801] req@ffff9811d0bdc700 x1876350913563264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431801.638346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913563264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431801.638353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431801.638356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431801.638359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431801.638377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431801.638398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431801.638409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431801/real 1789431801] req@ffff9811d0bdd880 x1876350913563392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431801.638419:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913563392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431801.638428:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431801.638431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431801.638434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431802.533994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431803.557964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431804.581964:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431805.605986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431806.629997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431806.757924:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431806.757935:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431806.757939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431806.757949:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4176 00000100:00080000:1.0:1789431806.757964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431806.757979:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431806.758012:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913563520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431806.758039:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431806.758042:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431806.758043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431806.758047:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4176 00000100:00080000:1.0:1789431806.758050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431806.758053:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431806.758060:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431806.758072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913563648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431806.758082:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431806.758086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431806.758088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431806.758093:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4176 00000100:00100000:2.0:1789431806.758094:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913563520:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789431806.758096:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431806.758104:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431806.758109:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431806.758120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913563776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431806.758128:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431806.758130:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431806.758131:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431806.758134:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4176 00000100:00080000:1.0:1789431806.758136:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431806.758139:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431806.758142:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431806.758151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913563904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431806.758160:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431806.758196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431806/real 1789431806] req@ffff9811d0bde300 x1876350913563520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431806.758208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913563520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431806.758216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431806.758218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431806.758221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913563520:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431806.758225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431806.758240:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431806.758247:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431806/real 1789431806] req@ffff9811d0bdf100 x1876350913563648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431806.758254:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913563648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431806.758258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431806.758260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431806.758262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431806.758277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431806.758285:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431806.758290:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431806/real 1789431806] req@ffff9811d0bdf800 x1876350913563776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431806.758300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913563776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431806.758305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431806.758306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431806.758308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431806.758311:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563904:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431806.758319:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431806.758336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431806/real 1789431806] req@ffff9811d0bdd880 x1876350913563904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431806.758344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913563904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431806.758350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431806.758351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431806.758353:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913563904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431807.653980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431808.677978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431809.701964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431810.725979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431811.749981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431811.877952:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431811.877961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431811.877965:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431811.877974:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4181 00000100:00080000:3.0:1789431811.877981:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431811.877991:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431811.878022:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913564032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431811.878045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431811.878050:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431811.878052:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431811.878056:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4181 00000100:00080000:3.0:1789431811.878059:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431811.878063:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431811.878071:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431811.878083:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913564160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431811.878093:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913564032:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431811.878095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431811.878100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431811.878102:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431811.878105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4181 00000100:00080000:3.0:1789431811.878108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431811.878111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431811.878116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431811.878133:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913564288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431811.878141:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431811.878145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431811.878149:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789431811.878150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431811.878154:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4181 00000100:00080000:3.0:1789431811.878157:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431811.878161:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431811.878166:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431811.878174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913564416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431811.878191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431811/real 1789431811] req@ffff9811d1bc7480 x1876350913564032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431811.878209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913564032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431811.878217:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431811.878219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431811.878221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913564032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431811.878238:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431811.878252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431811.878260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431811/real 1789431811] req@ffff9811d1bc6a00 x1876350913564160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431811.878267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913564160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431811.878272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431811.878274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431811.878275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431811.878278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431811.878287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431811.878307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431811/real 1789431811] req@ffff9811d1bc5180 x1876350913564288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431811.878314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913564288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431811.878323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431811.878324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431811.878326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431811.878329:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431811.878341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431811.878347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431811/real 1789431811] req@ffff9811d1bc5c00 x1876350913564416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431811.878353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913564416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431811.878358:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431811.878359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431811.878361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431812.774025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431813.797978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431814.821942:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431815.845996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431816.869991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431816.997988:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431816.997998:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431816.998002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431816.998012:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4186 00000100:00080000:0.0:1789431816.998020:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431816.998031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431816.998068:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913564544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431816.998090:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431816.998092:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431816.998094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431816.998097:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4186 00000100:00080000:0.0:1789431816.998099:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431816.998101:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431816.998110:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431816.998118:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913564672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431816.998129:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431816.998132:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431816.998133:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431816.998134:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4186 00000100:00080000:0.0:1789431816.998136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431816.998138:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431816.998141:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431816.998148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913564800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431816.998168:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431816.998175:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431816.998177:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789431816.998177:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913564544:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431816.998191:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4186 00000100:00080000:0.0:1789431816.998194:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431816.998197:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431816.998201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431816.998211:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913564928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431816.998249:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431816.998299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431816/real 1789431816] req@ffff9811c8cd5c00 x1876350913564544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431816.998313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913564544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431816.998325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431816.998328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431816.998331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913564544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431816.998338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431816.998359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431816.998372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431816/real 1789431816] req@ffff9811c8cd4700 x1876350913564672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431816.998383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913564672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431816.998390:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431816.998393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431816.998397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431816.998420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431816.998434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431816.998441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431816/real 1789431816] req@ffff9811c8cd6680 x1876350913564800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431816.998448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913564800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431816.998453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431816.998455:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431816.998457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431816.998463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431816.998479:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431816.998489:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431816/real 1789431816] req@ffff9811c8cd7100 x1876350913564928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431816.998501:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913564928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431816.998514:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431816.998517:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431816.998520:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913564928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431817.894048:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431818.918027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431819.941988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431820.965984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431821.989979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431822.117918:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431822.117926:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431822.117931:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431822.117939:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4191 00000100:00080000:0.0:1789431822.117945:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431822.117954:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431822.117986:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913565056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431822.118019:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431822.118022:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431822.118024:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431822.118028:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4191 00000100:00080000:0.0:1789431822.118031:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431822.118045:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431822.118055:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431822.118057:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913565056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789431822.118070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913565184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431822.118089:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431822.118093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431822.118095:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431822.118099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4191 00000100:00080000:0.0:1789431822.118104:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789431822.118104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789431822.118112:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431822.118130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431822.118134:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431822/real 1789431822] req@ffff9811c8cd4000 x1876350913565056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431822.118146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913565312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431822.118154:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913565056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431822.118184:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789431822.118187:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431822.118189:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431822.118195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913565056:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431822.118196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431822.118201:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431822.118206:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4191 00000100:00080000:0.0:1789431822.118210:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431822.118216:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431822.118230:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565184:192.168.203.128@tcp:38:kworker.0 00000008:00100000:0.0:1789431822.118232:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431822.118257:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913565440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431822.118261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431822.118280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431822/real 1789431822] req@ffff9811c8cd7800 x1876350913565184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431822.118294:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913565184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431822.118302:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431822.118306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431822.118309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431822.118314:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431822.118328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431822.118335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431822/real 1789431822] req@ffff9811c8cd5180 x1876350913565312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431822.118341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913565312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431822.118346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431822.118348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431822.118349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431822.118357:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565440:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431822.118367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431822.118372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431822/real 1789431822] req@ffff9811c8cd7480 x1876350913565440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431822.118382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913565440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431822.118387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431822.118389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431822.118390:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431823.013989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431824.037983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431825.062002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431826.085965:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431827.110026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431827.237952:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431827.237962:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431827.237966:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431827.237976:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4196 00000100:00080000:2.0:1789431827.237984:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431827.237994:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431827.238029:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913565568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431827.238054:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431827.238058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431827.238060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431827.238063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4196 00000100:00080000:2.0:1789431827.238066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431827.238070:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431827.238080:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431827.238091:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913565696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431827.238097:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913565568:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431827.238108:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431827.238111:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431827.238113:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431827.238118:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4196 00000100:00080000:2.0:1789431827.238121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431827.238125:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431827.238129:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431827.238150:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913565824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431827.238152:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431827.238164:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431827.238168:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431827.238170:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431827.238174:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4196 00000100:00080000:2.0:1789431827.238178:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431827.238182:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431827.238200:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431827.238206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431827/real 1789431827] req@ffff9811d2101c00 x1876350913565568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431827.238213:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913565952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431827.238230:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913565568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431827.238241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431827.238245:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431827.238249:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913565568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431827.238268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431827.238288:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431827.238299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431827/real 1789431827] req@ffff9811d2103b80 x1876350913565696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431827.238309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913565696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431827.238316:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431827.238318:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431827.238320:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431827.238323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431827.238339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431827.238349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431827/real 1789431827] req@ffff9811d2102300 x1876350913565824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431827.238359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913565824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431827.238367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431827.238370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431827.238373:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431827.238378:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431827.238394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431827.238401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431827/real 1789431827] req@ffff9811d2102a00 x1876350913565952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431827.238408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913565952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431827.238413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431827.238415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431827.238417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913565952:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431828.133975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431829.157939:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431830.181974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431831.205980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431832.230026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431832.357933:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431832.357943:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431832.357948:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431832.357958:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4201 00000100:00080000:2.0:1789431832.357966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431832.357976:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431832.358011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913566080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431832.358037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431832.358041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431832.358043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431832.358048:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4201 00000100:00080000:2.0:1789431832.358051:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431832.358056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431832.358065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431832.358080:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913566208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431832.358087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913566080:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431832.358093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431832.358098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431832.358100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431832.358104:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4201 00000100:00080000:2.0:1789431832.358108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431832.358113:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431832.358118:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431832.358130:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913566336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431832.358141:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431832.358144:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431832.358146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789431832.358160:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431832.358164:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4201 00000100:00080000:2.0:1789431832.358169:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431832.358175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431832.358181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431832.358194:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913566464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431832.358206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431832/real 1789431832] req@ffff9811d2101500 x1876350913566080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431832.358225:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913566080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431832.358233:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431832.358235:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431832.358238:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913566080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431832.358260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431832.358282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431832.358294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431832/real 1789431832] req@ffff9811d2100000 x1876350913566208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431832.358306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913566208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431832.358314:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431832.358317:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431832.358320:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431832.358326:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431832.358341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431832.358348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431832/real 1789431832] req@ffff9811d2103800 x1876350913566336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431832.358354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913566336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431832.358359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431832.358361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431832.358362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431832.358366:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566464:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431832.358376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431832.358381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431832/real 1789431832] req@ffff9811d2102d80 x1876350913566464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431832.358397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913566464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431832.358403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431832.358404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431832.358407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566464:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431833.254011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431834.277953:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431835.301980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431836.325941:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431837.349944:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431837.477912:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431837.477922:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431837.477926:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431837.477934:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4207 00000100:00080000:3.0:1789431837.477941:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431837.477951:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431837.477982:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913566592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431837.478007:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431837.478010:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431837.478012:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431837.478016:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4207 00000100:00080000:3.0:1789431837.478019:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431837.478024:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431837.478034:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431837.478047:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913566592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431837.478049:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913566720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431837.478062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431837.478065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431837.478067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431837.478070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4207 00000100:00080000:3.0:1789431837.478075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431837.478080:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431837.478085:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789431837.478096:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431837.478101:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913566848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431837.478126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431837.478130:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431837.478132:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431837.478137:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4207 00000100:00100000:1.0:1789431837.478141:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431837/real 1789431837] req@ffff9811d1bc7800 x1876350913566592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431837.478141:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431837.478159:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913566592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431837.478159:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431837.478170:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789431837.478171:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431837.478176:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431837.478180:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913566592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431837.478189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913566976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431837.478199:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431837.478219:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431837.478231:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431837/real 1789431837] req@ffff9811d1bc4000 x1876350913566720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431837.478241:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913566720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431837.478249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431837.478251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431837.478254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431837.478259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431837.478274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431837.478284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431837/real 1789431837] req@ffff9811d1bc5880 x1876350913566848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431837.478293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913566848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431837.478299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431837.478302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431837.478304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431837.478315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566976:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431837.478333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431837.478342:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431837/real 1789431837] req@ffff9811d1bc5f80 x1876350913566976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431837.478350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913566976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431837.478362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431837.478365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431837.478367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913566976:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431838.373966:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431839.397991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431840.421988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431841.445983:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431842.470045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431842.597911:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431842.597919:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431842.597923:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431842.597932:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4212 00000100:00080000:3.0:1789431842.597940:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431842.597950:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431842.597983:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913567104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431842.598011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431842.598018:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431842.598020:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431842.598023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4212 00000100:00080000:3.0:1789431842.598027:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431842.598031:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431842.598040:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431842.598053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913567232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431842.598054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913567104:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431842.598068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431842.598071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431842.598072:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431842.598076:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4212 00000100:00080000:3.0:1789431842.598089:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431842.598092:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431842.598096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431842.598105:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913567360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431842.598113:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431842.598115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431842.598117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431842.598120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4212 00000100:00080000:3.0:1789431842.598123:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431842.598139:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431842.598143:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431842.598148:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431842.598152:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913567488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431842.598194:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431842/real 1789431842] req@ffff9811d1bc7100 x1876350913567104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431842.598221:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913567104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431842.598241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431842.598244:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431842.598248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913567104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431842.598298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431842.598342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431842.598351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431842/real 1789431842] req@ffff9811d1bc5500 x1876350913567232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431842.598360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913567232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431842.598366:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431842.598369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431842.598372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431842.598397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431842.598449:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431842.598459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431842/real 1789431842] req@ffff9811d1bc4700 x1876350913567360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431842.598479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913567360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431842.598484:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431842.598486:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431842.598489:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431842.598493:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567488:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431842.598518:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431842.598525:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431842/real 1789431842] req@ffff9811d1bc6680 x1876350913567488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431842.598542:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913567488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431842.598548:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431842.598550:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431842.598553:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567488:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431843.494031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431844.518021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431845.541975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431846.565990:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431847.590003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431847.717911:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431847.717918:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431847.717922:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431847.717930:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4217 00000100:00080000:3.0:1789431847.717937:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431847.717946:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431847.717977:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913567616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431847.718002:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431847.718004:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431847.718008:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431847.718012:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4217 00000100:00080000:3.0:1789431847.718014:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431847.718018:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431847.718025:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431847.718036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913567744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431847.718052:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913567616:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431847.718058:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431847.718062:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431847.718063:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431847.718066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4217 00000100:00080000:3.0:1789431847.718070:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431847.718073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431847.718077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431847.718086:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913567872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431847.718098:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431847.718100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431847.718101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789431847.718106:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431847.718107:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4217 00000100:00080000:3.0:1789431847.718111:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431847.718115:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431847.718119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431847.718126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913568000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431847.718139:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431847/real 1789431847] req@ffff9811d1bc6300 x1876350913567616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431847.718152:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913567616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431847.718160:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431847.718162:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431847.718165:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913567616:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431847.718180:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431847.718195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431847.718206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431847/real 1789431847] req@ffff9811d1bc6680 x1876350913567744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431847.718218:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913567744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431847.718225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431847.718229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431847.718232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431847.718238:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431847.718255:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431847.718265:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431847/real 1789431847] req@ffff9811d1bc4700 x1876350913567872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431847.718276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913567872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431847.718300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431847.718302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431847.718304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913567872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431847.718308:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568000:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431847.718323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431847.718330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431847/real 1789431847] req@ffff9811d1bc5500 x1876350913568000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431847.718337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913568000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431847.718342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431847.718343:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431847.718345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568000:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431848.613996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431849.637981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431850.661975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431851.685971:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431852.709989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431852.837910:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431852.837918:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431852.837921:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431852.837929:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4222 00000100:00080000:3.0:1789431852.837935:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431852.837943:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431852.837978:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913568128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431852.837996:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431852.838000:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431852.838001:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431852.838004:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4222 00000100:00080000:3.0:1789431852.838006:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431852.838021:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431852.838029:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431852.838038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913568256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431852.838047:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431852.838049:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913568128:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431852.838050:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431852.838054:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431852.838058:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4222 00000100:00080000:3.0:1789431852.838060:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431852.838062:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431852.838065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431852.838072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913568384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431852.838078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431852.838081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431852.838082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431852.838084:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4222 00000100:00080000:3.0:1789431852.838086:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431852.838088:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431852.838092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431852.838099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913568512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431852.838106:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431852.838136:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431852/real 1789431852] req@ffff9811d1bc7100 x1876350913568128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431852.838148:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913568128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431852.838155:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431852.838157:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431852.838160:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913568128:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431852.838174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431852.838186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431852.838193:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431852/real 1789431852] req@ffff9811d1bc5f80 x1876350913568256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431852.838200:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913568256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431852.838204:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431852.838206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431852.838208:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431852.838211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431852.838219:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431852.838224:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431852/real 1789431852] req@ffff9811d1bc5880 x1876350913568384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431852.838230:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913568384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431852.838235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431852.838236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431852.838238:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431852.838241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568512:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431852.838250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431852.838254:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431852/real 1789431852] req@ffff9811d1bc4000 x1876350913568512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431852.838261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913568512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431852.838265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431852.838267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431852.838268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568512:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431853.733955:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431854.758047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431855.782714:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431856.805997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431857.829993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431857.957963:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431857.957985:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431857.957990:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431857.958004:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4227 00000100:00080000:3.0:1789431857.958015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431857.958027:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431857.958065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913568640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431857.958095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431857.958098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431857.958099:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431857.958102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4227 00000100:00080000:3.0:1789431857.958104:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431857.958107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431857.958117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431857.958126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913568768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431857.958133:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431857.958136:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431857.958137:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431857.958140:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4227 00000100:00080000:3.0:1789431857.958143:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431857.958146:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431857.958149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431857.958157:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913568896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431857.958159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913568640:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431857.958165:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431857.958168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431857.958170:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431857.958171:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4227 00000100:00080000:3.0:1789431857.958173:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431857.958175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431857.958177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431857.958185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913569024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431857.958232:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431857.958286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431857/real 1789431857] req@ffff9811d1bc7800 x1876350913568640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431857.958302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913568640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431857.958312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431857.958322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431857.958330:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913568640:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431857.958337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431857.958360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431857.958370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431857/real 1789431857] req@ffff9811d1bc5c00 x1876350913568768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431857.958378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913568768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431857.958388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431857.958390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431857.958392:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431857.958413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431857.958424:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431857.958430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431857/real 1789431857] req@ffff9811d1bc5180 x1876350913568896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431857.958437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913568896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431857.958442:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431857.958444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431857.958446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913568896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431857.958449:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569024:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431857.958466:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431857.958476:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431857/real 1789431857] req@ffff9811d1bc6a00 x1876350913569024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431857.958488:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913569024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431857.958497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431857.958499:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431857.958503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569024:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431858.854021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431859.878074:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431860.902055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431861.926059:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431862.950002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431863.077928:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431863.077951:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431863.077955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431863.077966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4232 00000100:00080000:3.0:1789431863.077975:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431863.077986:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431863.078018:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913569152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431863.078046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431863.078052:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431863.078054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431863.078058:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4232 00000100:00080000:3.0:1789431863.078061:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431863.078066:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431863.078073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431863.078084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913569280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431863.078092:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913569152:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431863.078093:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431863.078100:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431863.078102:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431863.078105:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4232 00000100:00080000:3.0:1789431863.078107:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431863.078109:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431863.078112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431863.078120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913569408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431863.078126:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431863.078129:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431863.078130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431863.078132:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4232 00000100:00080000:3.0:1789431863.078134:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431863.078137:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431863.078140:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431863.078147:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913569536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431863.078193:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431863.078218:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431863.078231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569536:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431863.974023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431864.998065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431866.022014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789431866.150152:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789431866.150192:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789431866.150206:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431866.150256:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431866.150264:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431866.150272:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789431866.150829:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431863/real 1789431866] req@ffff9811d1bc6a00 x1876350913569280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431866.150880:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913569280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431866.150890:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431866.150894:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431866.150904:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431866.150913:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431863/real 1789431866] req@ffff9811d1bc5180 x1876350913569408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431866.150936:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913569408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431866.150943:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431866.150946:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431866.150949:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431866.150955:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431863/real 1789431866] req@ffff9811d1bc5c00 x1876350913569536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431866.150967:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913569536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431866.150975:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431866.150978:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431866.150981:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569536:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789431866.150987:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431863/real 1789431866] req@ffff9811d1bc7480 x1876350913569152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431866.150998:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913569152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431866.151005:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431866.151008:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431866.151011:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913569152:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789431867.045990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431868.070000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431868.197977:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431868.197985:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431868.197989:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431868.197997:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4237 00000100:00080000:0.0:1789431868.198004:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431868.198011:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431868.198034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913569664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431868.198054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431868.198058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431868.198060:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431868.198064:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4237 00000100:00080000:0.0:1789431868.198067:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431868.198072:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431868.198082:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431868.198095:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913569792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431868.198104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431868.198107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431868.198109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431868.198112:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4237 00000100:00080000:0.0:1789431868.198115:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431868.198119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913569664:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431868.198122:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431868.198131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431868.198143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913569920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431868.198156:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431868.198159:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431868.198161:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431868.198164:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4237 00000100:00080000:0.0:1789431868.198178:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431868.198183:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431868.198189:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431868.198189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789431868.198208:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913570048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431868.198244:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431868/real 1789431868] req@ffff9811c8cd6300 x1876350913569664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431868.198263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913569664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431868.198274:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431868.198277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431868.198281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913569664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431868.198302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431868.198332:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431868.198346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431868/real 1789431868] req@ffff9811c8cd7b80 x1876350913569792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431868.198358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913569792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431868.198365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431868.198369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431868.198377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431868.198384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431868.198404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431868.198415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431868/real 1789431868] req@ffff9811c8cd6a00 x1876350913569920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431868.198426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913569920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431868.198434:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431868.198437:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431868.198440:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913569920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431868.198445:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570048:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431868.198464:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431868.198475:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431868/real 1789431868] req@ffff9811c8cd4a80 x1876350913570048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431868.198488:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913570048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431868.198496:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431868.198499:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431868.198502:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570048:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431869.093962:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431870.117999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431871.141995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431872.166010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431873.190034:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431873.317988:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431873.317998:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431873.318002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431873.318012:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4242 00000100:00080000:3.0:1789431873.318021:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431873.318032:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431873.318066:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913570176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431873.318103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431873.318107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431873.318109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431873.318113:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4242 00000100:00080000:3.0:1789431873.318117:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431873.318122:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431873.318131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431873.318146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913570304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431873.318150:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913570176:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431873.318166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431873.318169:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431873.318171:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431873.318175:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4242 00000100:00080000:3.0:1789431873.318178:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431873.318192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431873.318200:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789431873.318211:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789431873.318213:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913570432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431873.318225:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431873.318228:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431873.318230:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431873.318234:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4242 00000100:00080000:3.0:1789431873.318238:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431873.318244:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431873/real 1789431873] req@ffff9811d1bc7800 x1876350913570176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431873.318245:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431873.318254:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431873.318261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913570176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431873.318268:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913570560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431873.318271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431873.318277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431873.318283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913570176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431873.318302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431873.318328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431873.318335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431873/real 1789431873] req@ffff9811d1bc4000 x1876350913570304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431873.318343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913570304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431873.318352:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431873.318355:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431873.318359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431873.318364:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431873.318380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431873.318390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431873/real 1789431873] req@ffff9811d1bc5880 x1876350913570432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431873.318409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913570432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431873.318417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431873.318419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431873.318421:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431873.318425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570560:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431873.318440:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431873.318446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431873/real 1789431873] req@ffff9811d1bc5f80 x1876350913570560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431873.318453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913570560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431873.318458:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431873.318459:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431873.318462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431874.213999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431875.238009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431876.262006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431877.285945:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431878.310012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431878.437904:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431878.437911:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431878.437915:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431878.437923:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4248 00000100:00080000:3.0:1789431878.437929:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431878.437937:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431878.437963:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913570688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431878.437980:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431878.437983:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431878.437984:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431878.437987:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4248 00000100:00080000:3.0:1789431878.437988:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431878.437990:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431878.437997:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431878.438004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913570816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431878.438011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431878.438013:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431878.438016:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789431878.438018:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913570688:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431878.438019:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4248 00000100:00080000:3.0:1789431878.438022:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431878.438027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431878.438031:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431878.438036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913570944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431878.438046:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431878.438049:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431878.438050:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431878.438052:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4248 00000100:00080000:3.0:1789431878.438053:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431878.438056:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431878.438058:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431878.438064:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913571072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431878.438080:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431878.438109:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431878/real 1789431878] req@ffff9811d1bc7100 x1876350913570688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431878.438121:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913570688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431878.438128:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431878.438131:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431878.438133:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913570688:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431878.438147:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431878.438161:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431878.438168:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431878/real 1789431878] req@ffff9811d1bc5500 x1876350913570816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431878.438175:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913570816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431878.438180:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431878.438182:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431878.438184:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431878.438187:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431878.438196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431878.438201:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431878/real 1789431878] req@ffff9811d1bc4700 x1876350913570944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431878.438208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913570944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431878.438212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431878.438214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431878.438215:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913570944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431878.438219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571072:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431878.438228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431878.438233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431878/real 1789431878] req@ffff9811d1bc6680 x1876350913571072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431878.438239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913571072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431878.438244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431878.438245:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431878.438247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571072:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431879.333944:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431880.358019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431881.382051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431882.406060:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431883.430036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431883.557964:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431883.557973:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431883.557978:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431883.557988:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4253 00000100:00080000:2.0:1789431883.557997:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431883.558008:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431883.558046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913571200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431883.558074:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431883.558077:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431883.558079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431883.558085:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4253 00000100:00080000:2.0:1789431883.558088:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431883.558094:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431883.558105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431883.558115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913571328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431883.558124:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431883.558127:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789431883.558128:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913571200:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431883.558129:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431883.558135:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4253 00000100:00080000:2.0:1789431883.558139:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431883.558142:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431883.558146:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431883.558154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913571456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431883.558161:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431883.558164:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431883.558165:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431883.558167:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4253 00000100:00080000:2.0:1789431883.558169:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431883.558171:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431883.558175:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431883.558181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913571584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431883.558201:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431883.558244:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431883/real 1789431883] req@ffff9811d2102d80 x1876350913571200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431883.558261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913571200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431883.558273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431883.558277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431883.558280:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913571200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431883.558305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431883.558333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431883.558345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431883/real 1789431883] req@ffff9811d2103800 x1876350913571328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431883.558356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913571328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431883.558362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431883.558365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431883.558368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431883.558373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431883.558391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431883.558400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431883/real 1789431883] req@ffff9811d2100000 x1876350913571456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431883.558409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913571456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431883.558416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431883.558419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431883.558421:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431883.558427:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571584:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431883.558446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431883.558455:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431883/real 1789431883] req@ffff9811d2101500 x1876350913571584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431883.558466:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913571584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431883.558474:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431883.558478:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431883.558480:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571584:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431884.454034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431885.478088:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431886.501970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431887.526001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431888.550007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431888.677955:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431888.677964:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431888.677969:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431888.677979:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4258 00000100:00080000:1.0:1789431888.677988:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431888.677999:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431888.678038:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913571712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431888.678061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431888.678064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431888.678065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431888.678068:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4258 00000100:00080000:1.0:1789431888.678070:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431888.678075:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431888.678085:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431888.678089:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913571712:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431888.678096:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913571840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431888.678104:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431888.678106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431888.678107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431888.678110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4258 00000100:00080000:1.0:1789431888.678111:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431888.678114:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431888.678117:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431888.678125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913571968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431888.678132:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431888.678134:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431888.678135:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431888.678137:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4258 00000100:00080000:1.0:1789431888.678139:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431888.678141:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431888.678145:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431888.678151:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913572096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431888.678157:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431888.678201:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431888/real 1789431888] req@ffff9811d0bdc700 x1876350913571712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431888.678220:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913571712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431888.678232:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431888.678236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431888.678239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913571712:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431888.678259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431888.678276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431888.678284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431888/real 1789431888] req@ffff9811d0bdc000 x1876350913571840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431888.678291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913571840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431888.678296:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431888.678298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431888.678300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431888.678304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431888.678314:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431888.678320:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431888/real 1789431888] req@ffff9811d0bded80 x1876350913571968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431888.678326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913571968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431888.678331:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431888.678333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431888.678334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913571968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431888.678338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572096:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431888.678347:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431888.678352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431888/real 1789431888] req@ffff9811d0bdf480 x1876350913572096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431888.678358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913572096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431888.678363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431888.678365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431888.678366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572096:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431889.573984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431890.597959:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431891.621980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431892.645957:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431893.669991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431893.797921:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431893.797930:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431893.797934:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431893.797941:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4263 00000100:00080000:0.0:1789431893.797948:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431893.797956:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431893.797987:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913572224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431893.798018:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431893.798021:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431893.798022:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431893.798025:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4263 00000100:00080000:0.0:1789431893.798028:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431893.798032:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431893.798039:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431893.798049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913572352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431893.798054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913572224:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431893.798058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431893.798062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431893.798063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431893.798066:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4263 00000100:00080000:0.0:1789431893.798068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431893.798070:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431893.798073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431893.798079:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913572480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431893.798085:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431893.798087:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431893.798088:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431893.798091:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4263 00000100:00080000:0.0:1789431893.798094:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431893.798096:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431893.798099:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431893.798107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913572608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789431893.798113:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431893.798150:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431893/real 1789431893] req@ffff9811c8cd5f80 x1876350913572224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431893.798166:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913572224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431893.798176:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431893.798180:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431893.798184:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913572224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431893.798201:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431893.798218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431893.798227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431893/real 1789431893] req@ffff9811c8cd6d80 x1876350913572352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431893.798238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913572352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431893.798245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431893.798248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431893.798251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431893.798256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431893.798268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431893.798275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431893/real 1789431893] req@ffff9811c8cd5500 x1876350913572480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431893.798291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913572480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431893.798299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431893.798301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431893.798304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431893.798308:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572608:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431893.798321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431893.798329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431893/real 1789431893] req@ffff9811c8cd4a80 x1876350913572608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431893.798339:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913572608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431893.798346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431893.798349:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431893.798351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572608:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431894.693946:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431895.718003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431896.741997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431897.766020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431898.790046:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431898.917970:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431898.917983:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431898.917988:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431898.918007:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4268 00000100:00080000:2.0:1789431898.918018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431898.918033:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431898.918081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913572736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431898.918115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431898.918119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431898.918121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431898.918126:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4268 00000100:00080000:2.0:1789431898.918130:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431898.918135:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431898.918146:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431898.918159:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913572864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431898.918172:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913572736:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431898.918172:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431898.918187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431898.918190:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431898.918194:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4268 00000100:00080000:2.0:1789431898.918198:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431898.918203:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431898.918209:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431898.918221:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913572992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431898.918232:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431898.918236:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431898.918239:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431898.918243:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4268 00000100:00080000:2.0:1789431898.918247:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431898.918251:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431898.918257:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789431898.918259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431898.918272:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913573120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431898.918321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431898/real 1789431898] req@ffff9811d2102a00 x1876350913572736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431898.918344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913572736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431898.918354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431898.918358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431898.918362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913572736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431898.918389:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431898.918413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431898.918426:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431898/real 1789431898] req@ffff9811d2102300 x1876350913572864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431898.918438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913572864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431898.918446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431898.918449:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431898.918453:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431898.918459:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431898.918478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431898.918488:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431898/real 1789431898] req@ffff9811d2103b80 x1876350913572992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431898.918508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913572992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431898.918515:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431898.918518:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431898.918521:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913572992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431898.918527:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573120:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431898.918549:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431898.918570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431898/real 1789431898] req@ffff9811d2101c00 x1876350913573120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431898.918583:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913573120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431898.918591:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431898.918595:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431898.918597:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573120:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431899.814025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431900.838089:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431901.862036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431902.886044:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431903.910015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431904.037983:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431904.037994:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431904.037999:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431904.038010:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4273 00000100:00080000:3.0:1789431904.038020:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431904.038032:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431904.038065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913573248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431904.038093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431904.038097:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431904.038099:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431904.038103:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4273 00000100:00080000:3.0:1789431904.038106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431904.038111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431904.038122:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431904.038135:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913573248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431904.038138:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913573376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431904.038152:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431904.038156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431904.038158:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431904.038162:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4273 00000100:00080000:3.0:1789431904.038166:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431904.038171:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431904.038176:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431904.038188:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913573504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431904.038192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431904.038203:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431904.038207:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431904.038209:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431904.038213:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4273 00000100:00080000:3.0:1789431904.038217:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431904.038222:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431904.038228:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431904.038232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431904/real 1789431904] req@ffff9811d1bc6300 x1876350913573248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431904.038241:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913573632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431904.038253:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913573248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431904.038269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431904.038273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431904.038277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913573248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431904.038300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431904.038321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431904.038332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431904/real 1789431904] req@ffff9811d1bc6680 x1876350913573376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431904.038342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913573376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431904.038349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431904.038351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431904.038354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431904.038358:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431904.038373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431904.038381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431904/real 1789431904] req@ffff9811d1bc4700 x1876350913573504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431904.038391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913573504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431904.038397:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431904.038400:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431904.038402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431904.038408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573632:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431904.038425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431904.038433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431904/real 1789431904] req@ffff9811d1bc5500 x1876350913573632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431904.038440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913573632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431904.038445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431904.038447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431904.038449:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573632:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431904.934009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431905.957994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431906.982026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431908.006009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431909.030005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789431909.157919:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431909.157929:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789431909.157933:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431909.157943:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4278 00000100:00080000:1.0:1789431909.157951:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431909.157966:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789431909.157995:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913573760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431909.158015:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431909.158017:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789431909.158019:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431909.158022:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4278 00000100:00080000:1.0:1789431909.158024:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431909.158029:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431909.158036:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431909.158042:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913573760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431909.158049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913573888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431909.158057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431909.158059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789431909.158061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789431909.158063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4278 00000100:00080000:1.0:1789431909.158065:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431909.158068:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431909.158070:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431909.158078:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913574016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431909.158084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789431909.158086:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789431909.158088:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789431909.158091:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789431909.158094:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4278 00000100:00080000:1.0:1789431909.158097:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431909.158100:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789431909.158104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431909.158111:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913574144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431909.158125:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431909/real 1789431909] req@ffff9811d0bdfb80 x1876350913573760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431909.158138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913573760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431909.158145:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431909.158147:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431909.158150:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913573760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431909.158173:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431909.158189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431909.158196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431909/real 1789431909] req@ffff9811d0bdea00 x1876350913573888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431909.158203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913573888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431909.158208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431909.158210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431909.158212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913573888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431909.158215:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431909.158223:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431909.158229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431909/real 1789431909] req@ffff9811d0bde680 x1876350913574016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431909.158235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913574016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431909.158239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431909.158241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431909.158242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431909.158245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574144:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431909.158254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431909.158259:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431909/real 1789431909] req@ffff9811d0bdd180 x1876350913574144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431909.158269:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913574144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431909.158274:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431909.158276:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431909.158277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574144:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431910.053992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431911.077975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431912.101975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431913.125979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431914.150035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431914.277968:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431914.277978:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431914.277983:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431914.277994:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4283 00000100:00080000:3.0:1789431914.278003:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431914.278013:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431914.278051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913574272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431914.278073:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431914.278076:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431914.278078:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431914.278081:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4283 00000100:00080000:3.0:1789431914.278089:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431914.278093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431914.278103:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431914.278116:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913574400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789431914.278117:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913574272:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431914.278144:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431914.278154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431914.278156:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431914.278161:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4283 00000100:00080000:3.0:1789431914.278181:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789431914.278188:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789431914.278190:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431914.278197:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431914.278209:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913574528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431914.278221:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431914.278224:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431914.278226:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789431914.278231:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431914/real 1789431914] req@ffff9811d1bc7100 x1876350913574272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789431914.278235:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4283 00000100:00080000:3.0:1789431914.278243:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431914.278257:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431914.278265:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431914.278270:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913574272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789431914.278282:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913574656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431914.278286:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431914.278303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431914.278310:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913574272:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431914.278338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431914.278357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431914.278366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431914/real 1789431914] req@ffff9811d1bc5f80 x1876350913574400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431914.278379:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913574400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431914.278384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431914.278386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431914.278388:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431914.278391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789431914.278408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431914.278415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431914/real 1789431914] req@ffff9811d1bc5880 x1876350913574528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431914.278423:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913574528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431914.278430:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431914.278433:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431914.278436:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789431914.278442:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574656:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789431914.278470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789431914.278480:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431914/real 1789431914] req@ffff9811d1bc4000 x1876350913574656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431914.278487:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913574656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431914.278496:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789431914.278498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789431914.278500:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574656:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431915.174054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431916.197974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431917.221985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431918.246037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431919.270047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431919.397980:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431919.397992:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431919.397997:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431919.398009:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4288 00000100:00080000:2.0:1789431919.398019:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431919.398032:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431919.398073:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913574784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431919.398102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431919.398106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431919.398108:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431919.398113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4288 00000100:00080000:2.0:1789431919.398116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431919.398120:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431919.398131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431919.398143:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913574912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431919.398154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431919.398156:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913574784:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431919.398157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431919.398160:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431919.398165:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4288 00000100:00080000:2.0:1789431919.398169:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431919.398173:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431919.398178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431919.398190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913575040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431919.398199:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431919.398202:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431919.398204:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431919.398207:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4288 00000100:00080000:2.0:1789431919.398211:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431919.398215:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431919.398219:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431919.398229:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913575168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431919.398237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431919.398286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431919/real 1789431919] req@ffff9811d2101f80 x1876350913574784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431919.398302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913574784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431919.398311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431919.398316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431919.398320:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913574784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431919.398344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431919.398366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431919.398377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431919/real 1789431919] req@ffff9811d2100e00 x1876350913574912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431919.398388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913574912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431919.398395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431919.398398:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431919.398401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913574912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431919.398406:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431919.398420:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431919.398429:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431919/real 1789431919] req@ffff9811d2101880 x1876350913575040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431919.398443:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913575040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431919.398451:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431919.398453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431919.398456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431919.398461:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575168:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431919.398476:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431919.398485:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431919/real 1789431919] req@ffff9811d2101180 x1876350913575168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431919.398496:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913575168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431919.398505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431919.398508:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431919.398510:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575168:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431920.293982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431921.317981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431922.341988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431923.366026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431924.390007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431924.517975:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431924.517985:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431924.517989:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431924.518000:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4294 00000100:00080000:0.0:1789431924.518009:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431924.518021:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431924.518073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913575296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431924.518100:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431924.518103:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431924.518104:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431924.518108:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4294 00000100:00080000:0.0:1789431924.518111:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431924.518117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431924.518129:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431924.518140:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913575296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789431924.518148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913575424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431924.518160:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431924.518164:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431924.518166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431924.518170:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4294 00000100:00080000:0.0:1789431924.518174:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431924.518180:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431924.518185:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431924.518198:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913575552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431924.518209:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789431924.518210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789431924.518216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431924.518218:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431924.518223:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4294 00000100:00080000:0.0:1789431924.518228:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431924.518234:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431924.518246:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431924.518251:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431924/real 1789431924] req@ffff9811c8cd6a00 x1876350913575296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789431924.518261:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913575680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431924.518269:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913575296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431924.518285:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431924.518288:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431924.518292:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913575296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431924.518312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431924.518326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431924.518335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431924/real 1789431924] req@ffff9811c8cd7b80 x1876350913575424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431924.518342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913575424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431924.518347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431924.518349:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431924.518351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431924.518354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431924.518363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431924.518368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431924/real 1789431924] req@ffff9811c8cd6300 x1876350913575552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431924.518374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913575552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431924.518379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431924.518380:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431924.518382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431924.518386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575680:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431924.518396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431924.518401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431924/real 1789431924] req@ffff9811c8cd7480 x1876350913575680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431924.518407:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913575680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431924.518412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431924.518414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431924.518416:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575680:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431925.414037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431926.438061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431927.461970:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431928.485980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431929.509985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431929.637936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431929.637943:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431929.637946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431929.637953:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4299 00000100:00080000:3.0:1789431929.637960:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431929.637970:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431929.638004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913575808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431929.638028:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431929.638032:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431929.638033:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431929.638037:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4299 00000100:00080000:3.0:1789431929.638040:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431929.638046:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431929.638055:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431929.638068:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913575808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431929.638071:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913575936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431929.638085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431929.638089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431929.638090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431929.638095:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4299 00000100:00080000:3.0:1789431929.638098:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431929.638103:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431929.638108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431929.638121:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913576064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431929.638130:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431929.638134:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431929.638135:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431929.638139:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4299 00000100:00080000:3.0:1789431929.638144:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431929.638149:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431929.638155:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431929.638162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789431929.638169:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913576192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431929.638202:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431929.638218:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431930.533969:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431931.557949:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431932.582021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789431932.710049:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789431932.710079:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789431932.710094:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431932.710126:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431932.710134:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789431932.710155:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789431932.710176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431929/real 1789431932] req@ffff9811d1bc7480 x1876350913575936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431932.710202:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913575936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431932.710212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431932.710217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431932.710220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913575936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431932.710227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431929/real 1789431932] req@ffff9811d1bc5c00 x1876350913576064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431932.710238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913576064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431932.710244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431932.710247:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431932.710249:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431932.710255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431929/real 1789431932] req@ffff9811d1bc7800 x1876350913575808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431932.710265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913575808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431932.710271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431932.710274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431932.710276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913575808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431932.710297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431929/real 1789431932] req@ffff9811d1bc5180 x1876350913576192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431932.710308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913576192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431932.710315:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431932.710318:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431932.710321:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431933.605967:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431934.630045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431934.757902:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431934.757914:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431934.757920:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431934.757931:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4304 00000100:00080000:3.0:1789431934.757941:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431934.757954:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431934.757993:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913576320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431934.758014:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431934.758017:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431934.758018:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431934.758022:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4304 00000100:00080000:3.0:1789431934.758024:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431934.758028:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431934.758037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431934.758047:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913576448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431934.758051:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913576320:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431934.758059:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431934.758062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431934.758063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431934.758066:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4304 00000100:00080000:3.0:1789431934.758068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431934.758071:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431934.758073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431934.758080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913576576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431934.758095:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431934.758099:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431934.758101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431934.758106:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4304 00000100:00080000:3.0:1789431934.758109:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431934.758114:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431934.758119:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431934.758131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913576704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431934.758137:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431934.758186:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431934/real 1789431934] req@ffff9811d1bc6a00 x1876350913576320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431934.758209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913576320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431934.758222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431934.758226:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431934.758231:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913576320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431934.758259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431934.758283:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431934.758296:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431934/real 1789431934] req@ffff9811d1bc5180 x1876350913576448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431934.758309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913576448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431934.758317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431934.758320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431934.758324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431934.758330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431934.758348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431934.758357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431934/real 1789431934] req@ffff9811d1bc7800 x1876350913576576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431934.758368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913576576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431934.758376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431934.758379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431934.758381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431934.758386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576704:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431934.758407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431934.758416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431934/real 1789431934] req@ffff9811d1bc5c00 x1876350913576704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431934.758427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913576704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431934.758435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431934.758438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431934.758441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576704:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789431935.653980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431936.678042:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431937.702030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789431938.725995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431939.750001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431939.877931:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431939.877941:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431939.877946:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431939.877956:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4309 00000100:00080000:2.0:1789431939.877965:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431939.877976:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431939.878017:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913576832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431939.878046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431939.878049:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431939.878050:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431939.878055:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4309 00000100:00080000:2.0:1789431939.878059:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431939.878066:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431939.878076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431939.878089:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913576832:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431939.878095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913576960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431939.878109:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431939.878112:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431939.878114:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431939.878117:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4309 00000100:00080000:2.0:1789431939.878120:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431939.878125:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431939.878129:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431939.878139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913577088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431939.878147:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431939.878149:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431939.878151:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789431939.878156:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431939.878158:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4309 00000100:00080000:2.0:1789431939.878162:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431939.878167:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431939.878171:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431939.878181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913577216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431939.878194:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431939/real 1789431939] req@ffff9811d2101180 x1876350913576832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431939.878215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913576832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431939.878228:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431939.878231:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431939.878234:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913576832:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431939.878251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431939.878266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431939.878274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431939/real 1789431939] req@ffff9811d2101880 x1876350913576960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431939.878282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913576960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431939.878287:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431939.878290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431939.878292:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913576960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431939.878295:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431939.878305:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431939.878311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431939/real 1789431939] req@ffff9811d2100e00 x1876350913577088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431939.878318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913577088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431939.878323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431939.878325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431939.878327:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431939.878331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577216:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431939.878341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431939.878347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431939/real 1789431939] req@ffff9811d2101f80 x1876350913577216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431939.878354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913577216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431939.878359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431939.878361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431939.878363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431940.774026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431941.798010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431942.822022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431943.845980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431944.869969:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431944.997931:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431944.997941:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431944.997946:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431944.997955:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4314 00000100:00080000:2.0:1789431944.997963:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431944.997974:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431944.998007:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913577344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431944.998033:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431944.998036:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431944.998038:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431944.998042:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4314 00000100:00080000:2.0:1789431944.998045:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431944.998049:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431944.998058:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431944.998072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913577472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431944.998073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913577344:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431944.998087:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431944.998090:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431944.998091:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431944.998094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4314 00000100:00080000:2.0:1789431944.998096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431944.998099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431944.998103:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431944.998110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913577600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431944.998121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431944.998123:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431944.998126:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789431944.998127:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431944.998130:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4314 00000100:00080000:2.0:1789431944.998133:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431944.998136:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431944.998140:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431944.998147:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913577728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431944.998161:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431944/real 1789431944] req@ffff9811d2101c00 x1876350913577344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431944.998174:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913577344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431944.998181:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431944.998184:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431944.998186:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913577344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431944.998209:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431944.998228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431944.998248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431944/real 1789431944] req@ffff9811d2103b80 x1876350913577472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431944.998261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913577472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431944.998269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431944.998271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431944.998274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431944.998277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431944.998291:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431944.998298:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431944/real 1789431944] req@ffff9811d2102300 x1876350913577600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431944.998305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913577600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431944.998310:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431944.998311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431944.998313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431944.998317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577728:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431944.998334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431944.998340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431944/real 1789431944] req@ffff9811d2102a00 x1876350913577728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431944.998347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913577728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431944.998356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431944.998358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431944.998360:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577728:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431945.893979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431946.917987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431947.941990:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431948.966006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431949.989975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431950.117910:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431950.117917:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431950.117921:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431950.117929:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4319 00000100:00080000:3.0:1789431950.117935:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431950.117943:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431950.117971:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913577856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431950.117990:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431950.117992:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431950.117993:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431950.117996:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4319 00000100:00080000:3.0:1789431950.117998:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431950.118000:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431950.118008:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431950.118016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913577984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431950.118023:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431950.118025:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431950.118026:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431950.118029:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4319 00000100:00080000:3.0:1789431950.118031:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431950.118033:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431950.118036:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431950.118042:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913577856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431950.118043:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913578112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431950.118056:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431950.118058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431950.118059:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431950.118061:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4319 00000100:00080000:3.0:1789431950.118063:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431950.118065:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431950.118068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431950.118075:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913578240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431950.118104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431950.118135:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431950/real 1789431950] req@ffff9811d1bc7480 x1876350913577856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431950.118149:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913577856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431950.118159:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431950.118163:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431950.118167:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913577856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431950.118173:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431950.118195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431950.118206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431950/real 1789431950] req@ffff9811d1bc4000 x1876350913577984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431950.118218:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913577984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431950.118226:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431950.118229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431950.118232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913577984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431950.118252:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431950.118271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431950.118282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431950/real 1789431950] req@ffff9811d1bc5880 x1876350913578112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431950.118293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913578112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431950.118300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431950.118303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431950.118306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431950.118312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578240:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431950.118329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431950.118352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431950/real 1789431950] req@ffff9811d1bc5f80 x1876350913578240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431950.118363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913578240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431950.118371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431950.118374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431950.118377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578240:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431951.013939:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431952.037980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431953.061928:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431954.086020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431955.110018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431955.237950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431955.237961:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431955.237965:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431955.237976:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4324 00000100:00080000:2.0:1789431955.237984:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431955.237996:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431955.238033:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913578368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431955.238063:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431955.238066:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431955.238068:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431955.238081:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4324 00000100:00080000:2.0:1789431955.238090:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431955.238096:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431955.238107:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431955.238109:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913578368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789431955.238127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913578496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431955.238141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431955.238144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431955.238147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431955.238151:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4324 00000100:00080000:2.0:1789431955.238157:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431955.238162:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789431955.238169:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789431955.238169:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431955.238185:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913578624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431955.238196:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431955.238200:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431955.238202:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431955.238207:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4324 00000100:00080000:2.0:1789431955.238212:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431955.238216:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431955/real 1789431955] req@ffff9811d2101500 x1876350913578368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789431955.238221:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431955.238231:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431955.238242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913578368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431955.238248:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913578752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789431955.238258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431955.238267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431955.238272:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913578368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431955.238298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431955.238339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431955.238354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431955/real 1789431955] req@ffff9811d2100000 x1876350913578496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431955.238364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913578496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431955.238370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431955.238372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431955.238375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431955.238379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431955.238393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431955.238400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431955/real 1789431955] req@ffff9811d2103800 x1876350913578624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431955.238408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913578624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431955.238417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431955.238419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431955.238421:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431955.238425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578752:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431955.238435:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431955.238440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431955/real 1789431955] req@ffff9811d2102d80 x1876350913578752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431955.238447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913578752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431955.238453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431955.238454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431955.238456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913578752:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431956.134013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431957.158002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431958.181966:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431959.205943:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431960.229927:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431960.357955:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431960.357966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431960.357971:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431960.357981:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4329 00000100:00080000:0.0:1789431960.357990:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431960.358002:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431960.358039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913578880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431960.358069:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431960.358073:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431960.358075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431960.358080:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4329 00000100:00080000:0.0:1789431960.358084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431960.358089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431960.358100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431960.358113:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913579008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431960.358119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913578880:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431960.358134:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431960.358149:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431960.358151:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431960.358156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4329 00000100:00080000:0.0:1789431960.358168:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431960.358173:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431960.358179:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789431960.358179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789431960.358195:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913579136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431960.358206:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431960.358210:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1789431960.358214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431960/real 1789431960] req@ffff9811c8cd5180 x1876350913578880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431960.358215:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431960.358225:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4329 00000100:00100000:1.0:1789431960.358236:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913578880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789431960.358239:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789431960.358249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789431960.358250:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00080000:1.0:1789431960.358256:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000008:00100000:0.0:1789431960.358261:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431960.358261:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913578880:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789431960.358302:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913579264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431960.358303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431960.358338:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431960.358351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431960/real 1789431960] req@ffff9811c8cd7800 x1876350913579008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431960.358361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913579008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431960.358367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431960.358369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431960.358372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431960.358376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431960.358389:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431960.358403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431960/real 1789431960] req@ffff9811c8cd4000 x1876350913579136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431960.358412:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913579136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431960.358417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431960.358420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431960.358422:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431960.358431:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579264:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431960.358453:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431960.358468:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431960/real 1789431960] req@ffff9811c8cd7100 x1876350913579264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431960.358482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913579264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431960.358491:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431960.358495:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431960.358498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579264:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431961.253989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431962.278012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431963.301997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431964.325999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431965.350015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431965.477930:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431965.477941:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431965.477946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431965.477957:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4335 00000100:00080000:0.0:1789431965.477969:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431965.477980:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431965.478012:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913579392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431965.478037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431965.478039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431965.478041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431965.478044:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4335 00000100:00080000:0.0:1789431965.478046:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431965.478049:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431965.478056:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431965.478065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913579520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431965.478072:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431965.478075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431965.478076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431965.478078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4335 00000100:00080000:0.0:1789431965.478081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431965.478085:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913579392:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431965.478089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431965.478094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431965.478108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913579648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431965.478127:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431965.478139:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431965.478141:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431965.478144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4335 00000100:00080000:0.0:1789431965.478148:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789431965.478152:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789431965.478156:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431965.478162:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431965.478173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913579776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431965.478191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431965/real 1789431965] req@ffff9811c8cd6680 x1876350913579392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431965.478212:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913579392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431965.478225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431965.478229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431965.478246:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913579392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431965.478253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431965.478272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431965.478281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431965/real 1789431965] req@ffff9811c8cd4700 x1876350913579520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431965.478290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913579520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431965.478300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431965.478302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431965.478305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431965.478322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431965.478333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431965.478340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431965/real 1789431965] req@ffff9811c8cd5c00 x1876350913579648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431965.478347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913579648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431965.478352:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431965.478354:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431965.478356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431965.478360:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579776:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431965.478370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431965.478376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431965/real 1789431965] req@ffff9811c8cd7100 x1876350913579776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431965.478392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913579776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431965.478398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431965.478400:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431965.478402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913579776:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431966.374045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431967.398008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431968.421998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431969.446079:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431970.469972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431970.597981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431970.597988:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431970.597992:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431970.597999:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4340 00000100:00080000:3.0:1789431970.598007:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431970.598015:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431970.598043:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913579904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431970.598071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431970.598075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431970.598076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431970.598080:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4340 00000100:00080000:3.0:1789431970.598082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431970.598085:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431970.598092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431970.598103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913580032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431970.598111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431970.598114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431970.598115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431970.598121:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4340 00000100:00080000:3.0:1789431970.598125:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789431970.598127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913579904:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431970.598130:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431970.598142:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431970.598150:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913580160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431970.598163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431970.598166:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431970.598167:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431970.598170:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4340 00000100:00080000:3.0:1789431970.598171:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431970.598175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431970.598178:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431970.598187:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913580288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431970.598225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431970.598283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431970/real 1789431970] req@ffff9811d1bc7100 x1876350913579904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431970.598300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913579904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431970.598312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431970.598317:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431970.598321:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913579904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431970.598349:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431970.598368:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431970.598377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431970/real 1789431970] req@ffff9811d1bc5500 x1876350913580032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431970.598385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913580032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431970.598391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431970.598393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431970.598395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431970.598399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431970.598409:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431970.598415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431970/real 1789431970] req@ffff9811d1bc4700 x1876350913580160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431970.598422:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913580160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431970.598426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431970.598428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431970.598430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431970.598433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580288:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431970.598443:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431970.598449:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431970/real 1789431970] req@ffff9811d1bc6680 x1876350913580288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431970.598455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913580288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789431995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431970.598460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431970.598462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431970.598463:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580288:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431971.494043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431972.517961:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431973.542065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431974.566044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431975.589995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789431975.717931:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431975.717951:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789431975.717956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431975.717965:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4345 00000100:00080000:0.0:1789431975.717977:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431975.717988:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789431975.718030:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913580416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431975.718052:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431975.718055:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789431975.718056:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431975.718059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4345 00000100:00080000:0.0:1789431975.718062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431975.718067:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431975.718078:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431975.718090:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913580544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431975.718095:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913580416:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789431975.718103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431975.718106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789431975.718108:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431975.718111:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4345 00000100:00080000:0.0:1789431975.718114:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431975.718117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431975.718120:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789431975.718127:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913580672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789431975.718134:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789431975.718137:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789431975.718138:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789431975.718141:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4345 00000100:00080000:0.0:1789431975.718145:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789431975.718148:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789431975.718151:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789431975.718160:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913580800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431975.718172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431975.718217:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431975/real 1789431975] req@ffff9811c8cd4000 x1876350913580416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431975.718231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913580416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431975.718239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431975.718241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431975.718244:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913580416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431975.718261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431975.718277:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431975.718285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431975/real 1789431975] req@ffff9811c8cd7800 x1876350913580544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431975.718291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913580544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431975.718296:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431975.718298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431975.718300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431975.718303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431975.718316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431975.718322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431975/real 1789431975] req@ffff9811c8cd5180 x1876350913580672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431975.718328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913580672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431975.718333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431975.718335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431975.718337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431975.718341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580800:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431975.718352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431975.718357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431975/real 1789431975] req@ffff9811c8cd7480 x1876350913580800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431975.718369:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913580800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431975.718375:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431975.718376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431975.718378:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913580800:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431976.614060:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431977.637993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431978.662001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431979.686054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431980.709980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431980.837956:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431980.837966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431980.837970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431980.837980:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4350 00000100:00080000:3.0:1789431980.837988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431980.837999:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431980.838036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913580928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431980.838060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431980.838062:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431980.838064:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431980.838066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4350 00000100:00080000:3.0:1789431980.838068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431980.838071:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431980.838081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431980.838091:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913581056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431980.838099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431980.838101:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431980.838104:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789431980.838105:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913580928:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789431980.838107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4350 00000100:00080000:3.0:1789431980.838112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431980.838116:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431980.838119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431980.838141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913581184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431980.838152:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431980.838156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431980.838158:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431980.838161:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4350 00000100:00080000:3.0:1789431980.838166:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431980.838171:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431980.838176:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431980.838187:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913581312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431980.838191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431980.838230:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431980/real 1789431980] req@ffff9811d1bc6300 x1876350913580928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431980.838243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913580928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431980.838251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431980.838253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431980.838256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913580928:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431980.838275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431980.838289:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431980.838297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431980/real 1789431980] req@ffff9811d1bc6680 x1876350913581056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431980.838304:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913581056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431980.838309:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431980.838311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431980.838312:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431980.838316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431980.838325:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431980.838337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431980/real 1789431980] req@ffff9811d1bc4700 x1876350913581184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431980.838345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913581184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431980.838349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431980.838351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431980.838353:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431980.838356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581312:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431980.838367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431980.838372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431980/real 1789431980] req@ffff9811d1bc5500 x1876350913581312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431980.838378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913581312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431980.838388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431980.838390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431980.838392:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581312:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431981.733999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431982.758011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431983.782003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431984.806089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431985.830022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789431985.957967:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431985.957977:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789431985.957983:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431985.957994:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4355 00000100:00080000:3.0:1789431985.958005:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431985.958017:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789431985.958070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913581440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431985.958108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431985.958112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789431985.958113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431985.958117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4355 00000100:00080000:3.0:1789431985.958121:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431985.958125:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431985.958136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789431985.958148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913581568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431985.958155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431985.958159:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789431985.958160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431985.958163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4355 00000100:00080000:3.0:1789431985.958165:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431985.958167:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431985.958170:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789431985.958180:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913581440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789431985.958180:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913581696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789431985.958200:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789431985.958203:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789431985.958204:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789431985.958208:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4355 00000100:00080000:3.0:1789431985.958210:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789431985.958212:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789431985.958216:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789431985.958224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913581824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789431985.958257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431985.958299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431985/real 1789431985] req@ffff9811d1bc7100 x1876350913581440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431985.958320:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913581440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431985.958329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431985.958333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431985.958336:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913581440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431985.958342:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431985.958363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431985.958376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431985/real 1789431985] req@ffff9811d1bc5f80 x1876350913581568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431985.958390:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913581568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431985.958399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431985.958403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431985.958405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431985.958429:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431985.958444:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431985.958452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431985/real 1789431985] req@ffff9811d1bc5880 x1876350913581696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431985.958460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913581696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431985.958465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431985.958467:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431985.958469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431985.958472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581824:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431985.958486:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431985.958496:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431985/real 1789431985] req@ffff9811d1bc4000 x1876350913581824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431985.958509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913581824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431985.958520:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431985.958523:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431985.958527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913581824:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431986.854058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431987.878009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431988.901997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431989.925963:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431990.950009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431991.077912:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431991.077919:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431991.077923:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431991.077930:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4360 00000100:00080000:2.0:1789431991.077937:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431991.077945:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431991.077974:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913581952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431991.077993:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431991.077995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431991.077997:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431991.078000:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4360 00000100:00080000:2.0:1789431991.078001:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431991.078004:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431991.078010:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431991.078019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913582080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431991.078028:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789431991.078030:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913581952:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431991.078032:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431991.078036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431991.078041:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4360 00000100:00080000:2.0:1789431991.078044:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431991.078049:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431991.078054:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431991.078077:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913582208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431991.078087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789431991.078091:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789431991.078092:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431991.078096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431991.078099:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4360 00000100:00080000:2.0:1789431991.078103:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431991.078111:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431991.078116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789431991.078123:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431991/real 1789431991] req@ffff9811d2102d80 x1876350913581952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789431991.078131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913582336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431991.078144:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913581952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431991.078153:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431991.078156:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431991.078160:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913581952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789431991.078176:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431991.078189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431991.078197:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431991/real 1789431991] req@ffff9811d2103800 x1876350913582080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431991.078204:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913582080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431991.078208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431991.078210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431991.078212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431991.078215:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789431991.078224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431991.078229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431991/real 1789431991] req@ffff9811d2100000 x1876350913582208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431991.078235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913582208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431991.078243:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431991.078245:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431991.078247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431991.078250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582336:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789431991.078258:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789431991.078263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431991/real 1789431991] req@ffff9811d2101500 x1876350913582336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431991.078269:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913582336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431991.078274:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431991.078275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431991.078277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582336:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431991.973941:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431992.997993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431994.022027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431995.045984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431996.069986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789431996.197938:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431996.197946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789431996.197950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431996.197958:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4365 00000100:00080000:2.0:1789431996.197965:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431996.197978:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789431996.198005:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913582464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431996.198033:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431996.198036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789431996.198037:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431996.198040:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4365 00000100:00080000:2.0:1789431996.198043:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431996.198047:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431996.198055:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431996.198065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913582592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431996.198067:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913582464:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789431996.198078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431996.198081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789431996.198082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431996.198084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4365 00000100:00080000:2.0:1789431996.198086:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431996.198088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431996.198092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789431996.198098:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913582720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789431996.198104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789431996.198107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789431996.198108:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789431996.198110:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4365 00000100:00080000:2.0:1789431996.198112:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789431996.198114:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789431996.198116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789431996.198122:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913582848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789431996.198169:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431996.198202:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431996.198225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582848:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789431997.094004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431998.118068:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789431999.141978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789431999.270044:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789431999.270074:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789431999.270084:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431999.270108:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431999.270114:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789431999.270120:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789431999.270136:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431996/real 1789431999] req@ffff9811d2102300 x1876350913582592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431999.270163:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913582592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431999.270173:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431999.270178:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431999.270181:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431999.270188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431996/real 1789431999] req@ffff9811d2103b80 x1876350913582720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431999.270199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913582720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431999.270206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431999.270209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431999.270212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789431999.270218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431996/real 1789431999] req@ffff9811d2101c00 x1876350913582848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431999.270229:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913582848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431999.270237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431999.270240:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431999.270243:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913582848:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789431999.270249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789431996/real 1789431999] req@ffff9811d2102a00 x1876350913582464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789431999.270257:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913582464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789431999.270264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789431999.270267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789431999.270270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913582464:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432000.166283:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432001.189973:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432001.317964:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432001.317978:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432001.317982:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432001.318004:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4370 00000100:00080000:2.0:1789432001.318018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432001.318033:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432001.318075:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913582976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432001.318110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432001.318113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432001.318115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432001.318118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4370 00000100:00080000:2.0:1789432001.318121:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432001.318125:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432001.318134:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432001.318143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913583104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432001.318153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432001.318157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432001.318159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432001.318163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4370 00000100:00080000:2.0:1789432001.318166:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432001.318172:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432001.318173:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913582976:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789432001.318177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432001.318189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913583232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432001.318200:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432001.318203:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432001.318205:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432001.318209:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4370 00000100:00080000:2.0:1789432001.318213:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432001.318218:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432001.318223:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432001.318234:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913583360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432001.318274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432001.318331:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432001/real 1789432001] req@ffff9811d2101f80 x1876350913582976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432001.318352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913582976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432001.318364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432001.318368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432001.318373:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913582976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432001.318382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432001.318405:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432001.318417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432001/real 1789432001] req@ffff9811d2100e00 x1876350913583104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432001.318429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913583104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432001.318437:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432001.318441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432001.318444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432001.318487:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432001.318508:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432001.318520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432001/real 1789432001] req@ffff9811d2101880 x1876350913583232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432001.318532:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913583232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432001.318540:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432001.318543:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432001.318547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432001.318553:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583360:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432001.318573:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432001.318584:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432001/real 1789432001] req@ffff9811d2101180 x1876350913583360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432001.318596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913583360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432001.318605:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432001.318609:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432001.318612:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583360:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432002.213968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432003.237943:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432004.261975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432005.286005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432006.309977:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432006.437919:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432006.437926:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432006.437930:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432006.437937:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4376 00000100:00080000:2.0:1789432006.437945:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432006.437963:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432006.437993:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913583488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432006.438012:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432006.438014:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432006.438015:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432006.438018:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4376 00000100:00080000:2.0:1789432006.438021:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432006.438024:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432006.438031:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432006.438039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913583616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432006.438046:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432006.438049:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432006.438050:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432006.438052:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913583488:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432006.438054:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4376 00000100:00080000:2.0:1789432006.438059:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432006.438062:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432006.438065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432006.438073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913583744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432006.438079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432006.438081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432006.438083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432006.438085:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4376 00000100:00080000:2.0:1789432006.438087:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432006.438090:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432006.438093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432006.438100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913583872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432006.438106:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432006.438141:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432006/real 1789432006] req@ffff9811d2101180 x1876350913583488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432006.438160:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913583488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432006.438169:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432006.438172:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432006.438174:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913583488:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432006.438189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432006.438209:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432006.438221:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432006/real 1789432006] req@ffff9811d2101880 x1876350913583616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432006.438239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913583616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432006.438246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432006.438249:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432006.438251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432006.438256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432006.438272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432006.438281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432006/real 1789432006] req@ffff9811d2100e00 x1876350913583744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432006.438291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913583744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432006.438303:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432006.438306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432006.438308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432006.438312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583872:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432006.438330:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432006.438339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432006/real 1789432006] req@ffff9811d2101f80 x1876350913583872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432006.438350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913583872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432006.438357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432006.438360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432006.438363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913583872:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432007.333958:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432008.358035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432009.382002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432010.405993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432011.429987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432011.557934:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432011.557946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432011.557950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432011.557962:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4381 00000100:00080000:2.0:1789432011.557971:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432011.557983:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432011.558025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913584000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432011.558050:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432011.558054:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432011.558056:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432011.558061:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4381 00000100:00080000:2.0:1789432011.558065:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432011.558070:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432011.558080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432011.558096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913584128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432011.558108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432011.558119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913584000:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432011.558126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432011.558130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432011.558136:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4381 00000100:00080000:2.0:1789432011.558140:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432011.558146:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432011.558153:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432011.558168:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913584256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432011.558179:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432011.558183:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432011.558185:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432011.558189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432011.558193:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4381 00000100:00080000:2.0:1789432011.558199:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432011.558204:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432011.558210:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432011.558224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102300 x1876350913584384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432011.558245:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432011/real 1789432011] req@ffff9811d2102a00 x1876350913584000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432011.558265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913584000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432011.558275:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432011.558279:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432011.558283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913584000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432011.558306:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432011.558331:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432011.558345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432011/real 1789432011] req@ffff9811d2101c00 x1876350913584128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432011.558358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913584128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432011.558367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432011.558371:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432011.558375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432011.558381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432011.558399:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432011.558409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432011/real 1789432011] req@ffff9811d2103b80 x1876350913584256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432011.558422:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913584256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432011.558431:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432011.558434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432011.558438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432011.558452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584384:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432011.558472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432011.558482:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432011/real 1789432011] req@ffff9811d2102300 x1876350913584384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432011.558495:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102300 x1876350913584384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432011.558505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432011.558508:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432011.558512:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584384:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432012.454023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432013.477996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432014.502014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432015.526005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432016.549998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432016.677955:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432016.677964:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432016.677967:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432016.677976:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4386 00000100:00080000:1.0:1789432016.677984:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432016.677992:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432016.678022:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913584512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432016.678044:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432016.678047:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432016.678048:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432016.678051:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4386 00000100:00080000:1.0:1789432016.678053:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432016.678056:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432016.678064:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432016.678072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913584640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432016.678081:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913584512:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432016.678093:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432016.678096:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432016.678097:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432016.678100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4386 00000100:00080000:1.0:1789432016.678103:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432016.678105:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432016.678108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432016.678117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913584768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432016.678123:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432016.678125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432016.678127:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432016.678130:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4386 00000100:00080000:1.0:1789432016.678132:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432016.678135:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432016.678138:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432016.678142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432016.678150:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913584896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432016.678179:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432016/real 1789432016] req@ffff9811d0bdc380 x1876350913584512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432016.678193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913584512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432016.678201:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432016.678205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432016.678207:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913584512:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432016.678224:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432016.678237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432016.678246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432016/real 1789432016] req@ffff9811d0bddc00 x1876350913584640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432016.678253:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913584640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432016.678259:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432016.678261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432016.678263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432016.678266:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432016.678276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432016.678281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432016/real 1789432016] req@ffff9811d0bdd500 x1876350913584768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432016.678288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913584768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432016.678293:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432016.678295:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432016.678297:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432016.678300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584896:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432016.678311:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432016.678316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432016/real 1789432016] req@ffff9811d0bdd180 x1876350913584896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432016.678323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913584896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432016.678329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432016.678330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432016.678332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913584896:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432017.574010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432018.598041:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432019.622022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432020.645980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432021.670121:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432021.797948:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432021.797957:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432021.797961:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432021.797969:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4391 00000100:00080000:3.0:1789432021.797976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432021.797983:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432021.798010:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913585024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432021.798035:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432021.798038:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432021.798040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432021.798044:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4391 00000100:00080000:3.0:1789432021.798047:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432021.798051:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432021.798060:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432021.798073:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913585152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432021.798081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789432021.798084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913585024:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432021.798084:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432021.798087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432021.798093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4391 00000100:00080000:3.0:1789432021.798096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432021.798099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432021.798102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432021.798111:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913585280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432021.798118:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432021.798120:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432021.798121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432021.798123:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4391 00000100:00080000:3.0:1789432021.798134:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432021.798137:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432021.798140:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432021.798145:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432021.798155:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913585408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432021.798182:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432021/real 1789432021] req@ffff9811d1bc7480 x1876350913585024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432021.798199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913585024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432021.798210:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432021.798213:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432021.798216:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913585024:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432021.798235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432021.798255:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432021.798266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432021/real 1789432021] req@ffff9811d1bc5c00 x1876350913585152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432021.798278:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913585152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432021.798285:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432021.798289:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432021.798292:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432021.798297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432021.798312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432021.798321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432021/real 1789432021] req@ffff9811d1bc7800 x1876350913585280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432021.798331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913585280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432021.798339:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432021.798342:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432021.798346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432021.798351:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585408:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432021.798366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432021.798375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432021/real 1789432021] req@ffff9811d1bc5180 x1876350913585408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432021.798391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913585408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432021.798399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432021.798402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432021.798405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585408:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432022.694016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432023.717978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432024.741969:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432025.765976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432026.789969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432026.917917:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432026.917925:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432026.917928:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432026.917937:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4396 00000100:00080000:3.0:1789432026.917943:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432026.917951:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432026.917979:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913585536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432026.917999:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432026.918003:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432026.918005:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432026.918009:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4396 00000100:00080000:3.0:1789432026.918012:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432026.918016:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432026.918025:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432026.918031:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913585536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432026.918038:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913585664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432026.918046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432026.918051:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432026.918053:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432026.918055:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4396 00000100:00080000:3.0:1789432026.918058:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432026.918062:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432026.918065:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432026.918082:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432026.918082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913585792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432026.918091:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432026.918094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432026.918095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432026.918098:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4396 00000100:00080000:3.0:1789432026.918101:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432026.918104:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432026.918108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432026.918115:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432026/real 1789432026] req@ffff9811d1bc6a00 x1876350913585536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432026.918116:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913585920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432026.918132:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913585536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432026.918141:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432026.918145:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432026.918148:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913585536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432026.918168:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432026.918185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432026.918193:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432026/real 1789432026] req@ffff9811d1bc5180 x1876350913585664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432026.918201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913585664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432026.918206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432026.918208:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432026.918210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432026.918213:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432026.918222:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432026.918227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432026/real 1789432026] req@ffff9811d1bc7800 x1876350913585792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432026.918233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913585792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432026.918237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432026.918239:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432026.918240:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432026.918243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432026.918253:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432026.918258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432026/real 1789432026] req@ffff9811d1bc5c00 x1876350913585920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432026.918264:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913585920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432026.918272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432026.918274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432026.918276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913585920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432027.813992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432028.837970:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432029.862071:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432030.885970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432031.910039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432032.037915:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432032.037924:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432032.037929:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432032.037939:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4401 00000100:00080000:1.0:1789432032.037951:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432032.037962:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432032.038004:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913586048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432032.038033:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432032.038036:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432032.038037:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432032.038041:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4401 00000100:00080000:1.0:1789432032.038043:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432032.038045:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432032.038055:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432032.038064:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913586176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432032.038072:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432032.038074:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432032.038075:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432032.038078:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4401 00000100:00080000:1.0:1789432032.038079:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432032.038081:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432032.038084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432032.038091:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913586304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432032.038101:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913586048:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432032.038102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432032.038115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432032.038118:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432032.038122:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4401 00000100:00080000:1.0:1789432032.038125:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432032.038127:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432032.038130:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432032.038143:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913586432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432032.038176:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432032.038209:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432032/real 1789432032] req@ffff9811d0bde680 x1876350913586048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432032.038222:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913586048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432032.038230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432032.038233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432032.038235:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913586048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432032.038241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432032.038254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432032.038261:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432032/real 1789432032] req@ffff9811d0bdea00 x1876350913586176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432032.038267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913586176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432032.038279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432032.038282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432032.038284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432032.038301:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432032.038312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432032.038318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432032/real 1789432032] req@ffff9811d0bdfb80 x1876350913586304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432032.038324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913586304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432032.038329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432032.038331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432032.038332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432032.038336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432032.038353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432032.038363:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432032/real 1789432032] req@ffff9811d0bdf480 x1876350913586432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432032.038374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913586432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432032.038382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432032.038385:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432032.038387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432032.934079:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432033.958039:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432034.982016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432036.005985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432037.029995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432037.157954:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432037.157966:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432037.157970:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432037.157982:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4406 00000100:00080000:3.0:1789432037.157992:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432037.158010:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432037.158048:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913586560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432037.158093:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432037.158097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432037.158099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432037.158104:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4406 00000100:00080000:3.0:1789432037.158108:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432037.158114:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432037.158123:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432037.158138:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913586688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432037.158151:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432037.158154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432037.158157:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789432037.158159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913586560:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432037.158161:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4406 00000100:00080000:3.0:1789432037.158167:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432037.158172:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432037.158176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432037.158186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913586816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432037.158193:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432037.158196:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432037.158197:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432037.158200:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4406 00000100:00080000:3.0:1789432037.158220:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432037.158224:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432037.158228:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432037.158230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432037.158238:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913586944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432037.158276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432037/real 1789432037] req@ffff9811d1bc7480 x1876350913586560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432037.158298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913586560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432037.158309:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432037.158314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432037.158318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913586560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432037.158340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432037.158365:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432037.158378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432037/real 1789432037] req@ffff9811d1bc4000 x1876350913586688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432037.158392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913586688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432037.158400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432037.158404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432037.158408:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432037.158414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432037.158434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432037.158446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432037/real 1789432037] req@ffff9811d1bc5880 x1876350913586816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432037.158459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913586816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432037.158467:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432037.158471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432037.158475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432037.158481:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586944:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432037.158502:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432037.158514:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432037/real 1789432037] req@ffff9811d1bc5f80 x1876350913586944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432037.158526:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913586944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432037.158536:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432037.158539:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432037.158543:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913586944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432038.054069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432039.078026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432040.102077:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432041.126029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432042.149985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432042.277925:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432042.277936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432042.277941:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432042.277951:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4411 00000100:00080000:2.0:1789432042.277960:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432042.277970:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432042.277999:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913587072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432042.278023:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432042.278026:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432042.278027:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432042.278030:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4411 00000100:00080000:2.0:1789432042.278033:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432042.278036:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432042.278043:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432042.278052:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913587200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432042.278060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432042.278064:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789432042.278066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913587072:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432042.278066:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432042.278074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4411 00000100:00080000:2.0:1789432042.278077:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432042.278080:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432042.278083:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432042.278091:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913587328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432042.278099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432042.278101:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432042.278103:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432042.278106:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4411 00000100:00080000:2.0:1789432042.278108:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432042.278111:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432042.278114:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432042.278122:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913587456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432042.278126:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432042.278162:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432042/real 1789432042] req@ffff9811d2101500 x1876350913587072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432042.278176:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913587072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432042.278184:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432042.278186:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432042.278189:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913587072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432042.278206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432042.278220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432042.278228:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432042/real 1789432042] req@ffff9811d2100000 x1876350913587200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432042.278236:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913587200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432042.278242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432042.278244:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432042.278247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432042.278250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432042.278267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432042.278278:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432042/real 1789432042] req@ffff9811d2103800 x1876350913587328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432042.278291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913587328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432042.278299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432042.278303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432042.278306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432042.278312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432042.278329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432042.278337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432042/real 1789432042] req@ffff9811d2102d80 x1876350913587456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432042.278344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913587456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432042.278351:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432042.278354:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432042.278358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432043.174017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432044.198027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432045.222022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432046.245994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432047.270047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432047.397947:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432047.397958:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432047.397963:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432047.397979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4416 00000100:00080000:3.0:1789432047.397988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432047.398001:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432047.398043:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913587584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432047.398070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432047.398074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432047.398076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432047.398080:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4416 00000100:00080000:3.0:1789432047.398084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432047.398090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432047.398099:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432047.398114:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913587712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432047.398123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913587584:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432047.398129:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432047.398137:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432047.398139:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432047.398143:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4416 00000100:00080000:3.0:1789432047.398147:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432047.398152:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432047.398156:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432047.398167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913587840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432047.398174:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432047.398177:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432047.398178:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432047.398181:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4416 00000100:00080000:3.0:1789432047.398183:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432047.398185:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432047.398189:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432047.398194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432047.398199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913587968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432047.398248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432047/real 1789432047] req@ffff9811d1bc7100 x1876350913587584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432047.398268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913587584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432047.398289:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432047.398294:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432047.398298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913587584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432047.398324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432047.398349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432047.398363:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432047/real 1789432047] req@ffff9811d1bc5500 x1876350913587712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432047.398377:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913587712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432047.398386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432047.398390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432047.398394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432047.398401:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432047.398421:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432047.398433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432047/real 1789432047] req@ffff9811d1bc4700 x1876350913587840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432047.398446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913587840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432047.398455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432047.398459:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432047.398462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432047.398469:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587968:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432047.398488:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432047.398500:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432047/real 1789432047] req@ffff9811d1bc6680 x1876350913587968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432047.398513:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913587968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432047.398522:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432047.398526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432047.398529:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913587968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432048.293998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432049.317989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432050.341963:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432051.365977:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432052.389933:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432052.517959:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432052.517970:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432052.517974:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432052.517983:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4422 00000100:00080000:0.0:1789432052.517991:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432052.518001:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432052.518040:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913588096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432052.518065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432052.518069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432052.518071:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432052.518075:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4422 00000100:00080000:0.0:1789432052.518078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432052.518084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432052.518092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432052.518105:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913588224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432052.518109:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913588096:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432052.518123:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432052.518127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432052.518128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432052.518133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4422 00000100:00080000:0.0:1789432052.518136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432052.518141:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432052.518146:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432052.518157:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913588352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432052.518173:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432052.518175:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432052.518179:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432052.518181:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432052.518186:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4422 00000100:00080000:0.0:1789432052.518189:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432052.518194:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432052.518199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432052.518210:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913588480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432052.518216:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432052/real 1789432052] req@ffff9811c8cd6300 x1876350913588096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432052.518242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913588096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432052.518250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432052.518253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432052.518256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913588096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432052.518272:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432052.518287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432052.518295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432052/real 1789432052] req@ffff9811c8cd7b80 x1876350913588224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432052.518303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913588224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432052.518307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432052.518309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432052.518311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432052.518314:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432052.518324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432052.518329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432052/real 1789432052] req@ffff9811c8cd6a00 x1876350913588352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432052.518335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913588352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432052.518340:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432052.518341:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432052.518343:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432052.518346:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432052.518356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432052.518360:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432052/real 1789432052] req@ffff9811c8cd4a80 x1876350913588480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432052.518367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913588480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432052.518371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432052.518373:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432052.518374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432053.413994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432054.438045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432055.462037:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432056.485996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432057.509953:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432057.637932:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432057.637942:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432057.637946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432057.637955:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4427 00000100:00080000:2.0:1789432057.637963:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432057.637973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432057.638002:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913588608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432057.638022:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432057.638024:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432057.638026:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432057.638028:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4427 00000100:00080000:2.0:1789432057.638031:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432057.638034:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432057.638041:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432057.638049:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913588736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432057.638057:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432057.638059:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432057.638060:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432057.638066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913588608:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432057.638066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4427 00000100:00080000:2.0:1789432057.638070:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432057.638075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432057.638078:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432057.638085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913588864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432057.638091:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432057.638094:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432057.638095:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432057.638097:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4427 00000100:00080000:2.0:1789432057.638099:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432057.638101:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432057.638104:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432057.638111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913588992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432057.638129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432057.638166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432057/real 1789432057] req@ffff9811d2102d80 x1876350913588608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432057.638181:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913588608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432057.638187:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432057.638190:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432057.638192:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913588608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432057.638216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432057.638233:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432057.638241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432057/real 1789432057] req@ffff9811d2103800 x1876350913588736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432057.638249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913588736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432057.638253:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432057.638255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432057.638257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432057.638261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432057.638270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432057.638275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432057/real 1789432057] req@ffff9811d2100000 x1876350913588864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432057.638282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913588864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432057.638286:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432057.638288:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432057.638289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432057.638292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432057.638301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432057.638306:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432057/real 1789432057] req@ffff9811d2101500 x1876350913588992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432057.638312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913588992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432057.638320:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432057.638322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432057.638324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913588992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432058.534002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432059.558001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432060.581982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432061.606064:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432062.630002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432062.757935:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432062.757944:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432062.757948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432062.757957:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4432 00000100:00080000:3.0:1789432062.757965:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432062.757973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432062.758002:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913589120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432062.758025:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432062.758028:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432062.758029:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432062.758032:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4432 00000100:00080000:3.0:1789432062.758034:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432062.758036:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432062.758043:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432062.758053:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913589248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432062.758061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432062.758062:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913589120:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432062.758064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432062.758067:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432062.758071:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4432 00000100:00080000:3.0:1789432062.758073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432062.758075:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432062.758078:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432062.758085:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913589376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432062.758092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432062.758094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432062.758095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432062.758097:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4432 00000100:00080000:3.0:1789432062.758100:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432062.758109:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432062.758112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432062.758120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913589504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432062.758142:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432062.758163:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432062.758181:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432063.653996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432064.678061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432065.701981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432065.830090:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432065.830123:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432065.830134:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432065.830171:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432065.830180:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432065.830187:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789432065.830445:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432062/real 1789432065] req@ffff9811d1bc6680 x1876350913589248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432065.830471:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913589248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432065.830482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432065.830488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432065.830492:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432065.830499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432062/real 1789432065] req@ffff9811d1bc4700 x1876350913589376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432065.830511:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913589376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432065.830519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432065.830528:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432065.830532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432065.830539:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432062/real 1789432065] req@ffff9811d1bc5500 x1876350913589504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432065.830563:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913589504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432065.830571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432065.830575:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432065.830578:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589504:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789432065.830584:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432062/real 1789432065] req@ffff9811d1bc6300 x1876350913589120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432065.830596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913589120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432065.830603:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432065.830606:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432065.830609:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913589120:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432066.725991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432067.749986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432067.877950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432067.877960:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432067.877964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432067.877973:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4437 00000100:00080000:3.0:1789432067.877983:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432067.877993:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432067.878025:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913589632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432067.878051:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432067.878055:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432067.878057:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432067.878062:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4437 00000100:00080000:3.0:1789432067.878065:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432067.878070:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432067.878079:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432067.878092:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913589760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432067.878102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432067.878109:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789432067.878111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913589632:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432067.878112:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432067.878120:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4437 00000100:00080000:3.0:1789432067.878123:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432067.878136:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432067.878140:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432067.878151:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913589888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432067.878161:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432067.878164:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432067.878165:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432067.878171:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4437 00000800:00000100:2.0:1789432067.878172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432067.878177:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432067.878183:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432067.878188:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432067.878199:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913590016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432067.878212:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432067/real 1789432067] req@ffff9811d1bc7100 x1876350913589632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432067.878226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913589632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432067.878233:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432067.878236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432067.878239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913589632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432067.878254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432067.878270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432067.878278:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432067/real 1789432067] req@ffff9811d1bc5f80 x1876350913589760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432067.878285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913589760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432067.878290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432067.878292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432067.878294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432067.878300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432067.878309:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432067.878315:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432067/real 1789432067] req@ffff9811d1bc5880 x1876350913589888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432067.878323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913589888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432067.878328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432067.878330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432067.878333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913589888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432067.878338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432067.878349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432067.878354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432067/real 1789432067] req@ffff9811d1bc4000 x1876350913590016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432067.878360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913590016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432067.878365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432067.878366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432067.878368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432068.773972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432069.797972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432070.821993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432071.845987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432072.869993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432072.997925:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432072.997935:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432072.997939:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432072.997948:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4442 00000100:00080000:1.0:1789432072.997956:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432072.997966:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432072.998000:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913590144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432072.998029:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432072.998033:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432072.998035:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432072.998039:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4442 00000100:00080000:1.0:1789432072.998044:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432072.998050:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432072.998059:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432072.998067:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913590144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432072.998072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913590272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432072.998084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432072.998087:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432072.998089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432072.998092:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4442 00000100:00080000:1.0:1789432072.998096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432072.998102:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432072.998107:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432072.998120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913590400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432072.998125:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432072.998133:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432072.998136:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432072.998138:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432072.998142:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4442 00000100:00080000:1.0:1789432072.998146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432072.998151:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432072.998156:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432072.998164:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432072/real 1789432072] req@ffff9811d0bded80 x1876350913590144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432072.998170:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913590528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432072.998191:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913590144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432072.998203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432072.998207:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432072.998210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913590144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432072.998228:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432072.998249:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432072.998260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432072/real 1789432072] req@ffff9811d0bdc000 x1876350913590272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432072.998272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913590272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432072.998280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432072.998283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432072.998287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432072.998292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432072.998310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432072.998321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432072/real 1789432072] req@ffff9811d0bdc700 x1876350913590400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432072.998338:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913590400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432072.998345:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432072.998348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432072.998351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432072.998356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590528:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432072.998374:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432072.998384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432072/real 1789432072] req@ffff9811d0bdd880 x1876350913590528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432072.998395:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913590528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432072.998402:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432072.998405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432072.998407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432073.894029:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432074.917983:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432075.941981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432076.966069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432077.989983:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432078.117899:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432078.117910:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432078.117914:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432078.117923:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4447 00000100:00080000:1.0:1789432078.117932:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432078.117942:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432078.117975:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913590656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432078.117995:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432078.117998:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432078.117999:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432078.118002:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4447 00000100:00080000:1.0:1789432078.118004:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432078.118007:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432078.118014:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432078.118023:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913590784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432078.118030:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432078.118035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789432078.118035:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913590656:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432078.118039:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432078.118044:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4447 00000100:00080000:1.0:1789432078.118046:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432078.118048:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432078.118051:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432078.118059:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913590912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432078.118066:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432078.118068:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432078.118069:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432078.118072:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4447 00000100:00080000:1.0:1789432078.118074:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432078.118076:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432078.118079:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432078.118087:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432078.118090:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913591040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432078.118120:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432078/real 1789432078] req@ffff9811d0bdf800 x1876350913590656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432078.118137:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913590656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432078.118144:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432078.118147:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432078.118149:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913590656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432078.118164:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432078.118176:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432078.118193:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432078/real 1789432078] req@ffff9811d0bdf100 x1876350913590784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432078.118201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913590784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432078.118209:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432078.118211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432078.118213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432078.118216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432078.118227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432078.118233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432078/real 1789432078] req@ffff9811d0bde300 x1876350913590912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432078.118240:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913590912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432078.118248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432078.118250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432078.118252:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913590912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432078.118255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432078.118263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432078.118268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432078/real 1789432078] req@ffff9811d0bdd880 x1876350913591040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432078.118274:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913591040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432078.118279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432078.118280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432078.118282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432079.013982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432080.037978:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432081.061996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432082.085949:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432083.109939:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432083.237924:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432083.237932:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432083.237935:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432083.237944:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4452 00000100:00080000:3.0:1789432083.237952:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432083.237961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432083.237990:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913591168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432083.238014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432083.238017:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432083.238019:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432083.238022:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4452 00000100:00080000:3.0:1789432083.238025:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432083.238030:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432083.238038:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432083.238050:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913591168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432083.238054:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913591296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432083.238064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432083.238067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432083.238069:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432083.238072:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4452 00000100:00080000:3.0:1789432083.238075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432083.238090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432083.238095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432083.238097:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432083.238110:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913591424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432083.238119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432083.238122:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432083.238124:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432083.238127:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4452 00000100:00080000:3.0:1789432083.238131:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432083.238135:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432083/real 1789432083] req@ffff9811d1bc7480 x1876350913591168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432083.238139:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432083.238146:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432083.238161:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913591168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432083.238168:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913591552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432083.238177:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432083.238184:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432083.238187:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913591168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432083.238204:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432083.238219:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432083.238227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432083/real 1789432083] req@ffff9811d1bc5c00 x1876350913591296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432083.238234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913591296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432083.238239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432083.238241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432083.238243:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432083.238246:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432083.238255:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432083.238260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432083/real 1789432083] req@ffff9811d1bc7800 x1876350913591424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432083.238268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913591424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432083.238275:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432083.238277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432083.238289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432083.238298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432083.238317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432083.238325:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432083/real 1789432083] req@ffff9811d1bc5180 x1876350913591552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432083.238332:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913591552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432083.238338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432083.238340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432083.238342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432084.133991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432085.157998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432086.181994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432087.205934:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432088.229991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432088.358016:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432088.358025:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432088.358029:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432088.358037:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4457 00000100:00080000:1.0:1789432088.358044:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432088.358052:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432088.358092:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913591680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432088.358117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432088.358120:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432088.358122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432088.358127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4457 00000100:00080000:1.0:1789432088.358131:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432088.358136:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432088.358146:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432088.358146:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913591680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432088.358161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913591808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432088.358173:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432088.358176:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432088.358179:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432088.358182:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4457 00000100:00080000:1.0:1789432088.358185:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432088.358190:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432088.358194:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432088.358195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432088.358207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913591936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432088.358218:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432088.358221:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432088.358222:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789432088.358226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432088/real 1789432088] req@ffff9811d0bdc700 x1876350913591680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432088.358229:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4457 00000100:00080000:1.0:1789432088.358233:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432088.358239:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432088.358245:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432088.358253:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913591680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432088.358254:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913592064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432088.358283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432088.358287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432088.358291:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913591680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432088.358307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432088.358333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432088.358341:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432088/real 1789432088] req@ffff9811d0bdc000 x1876350913591808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432088.358349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913591808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432088.358354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432088.358356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432088.358358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432088.358361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432088.358370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432088.358375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432088/real 1789432088] req@ffff9811d0bded80 x1876350913591936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432088.358381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913591936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432088.358398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432088.358400:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432088.358402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913591936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432088.358406:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592064:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432088.358419:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432088.358425:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432088/real 1789432088] req@ffff9811d0bdf480 x1876350913592064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432088.358432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913592064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432088.358440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432088.358442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432088.358444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432089.253963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432090.277978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432091.301951:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432092.325991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432093.349936:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432093.477923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432093.477930:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432093.477933:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432093.477940:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4463 00000100:00080000:3.0:1789432093.477947:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432093.477956:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432093.477987:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913592192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432093.478011:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432093.478014:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432093.478015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432093.478018:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4463 00000100:00080000:3.0:1789432093.478021:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432093.478025:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432093.478033:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432093.478041:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913592192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432093.478046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913592320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432093.478058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432093.478064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432093.478066:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432093.478070:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4463 00000100:00080000:3.0:1789432093.478084:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789432093.478092:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432093.478092:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432093.478099:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432093.478110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913592448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432093.478117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432093.478120:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432093.478121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789432093.478125:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432093/real 1789432093] req@ffff9811d1bc6a00 x1876350913592192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432093.478126:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4463 00000100:00080000:3.0:1789432093.478132:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432093.478138:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432093.478142:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432093.478146:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913592192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432093.478151:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913592576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432093.478158:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432093.478163:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432093.478167:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913592192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432093.478190:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432093.478204:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432093.478213:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432093/real 1789432093] req@ffff9811d1bc5180 x1876350913592320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432093.478237:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913592320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432093.478246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432093.478249:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432093.478253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432093.478259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432093.478275:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432093.478282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432093/real 1789432093] req@ffff9811d1bc7800 x1876350913592448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432093.478289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913592448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432093.478294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432093.478295:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432093.478297:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432093.478310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432093.478328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432093.478338:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432093/real 1789432093] req@ffff9811d1bc5c00 x1876350913592576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432093.478349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913592576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432093.478368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432093.478370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432093.478372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432094.373985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432095.398003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432096.421992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432097.446031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432098.470044:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432098.598028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432098.598050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432098.598056:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432098.598070:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4468 00000100:00080000:3.0:1789432098.598084:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432098.598098:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432098.598144:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913592704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432098.598176:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432098.598180:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432098.598182:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432098.598186:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4468 00000100:00080000:3.0:1789432098.598191:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432098.598196:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432098.598208:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432098.598226:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913592832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432098.598229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913592704:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432098.598254:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432098.598257:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432098.598259:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432098.598271:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4468 00000100:00080000:3.0:1789432098.598275:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432098.598296:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432098.598302:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432098.598304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432098.598336:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913592960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432098.598351:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432098.598354:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432098.598356:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432098.598360:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4468 00000100:00080000:3.0:1789432098.598367:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432098.598368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432098/real 1789432098] req@ffff9811d1bc7480 x1876350913592704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432098.598379:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432098.598401:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432098.598407:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913592704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432098.598420:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913593088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432098.598421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432098.598434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432098.598445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913592704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432098.598478:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432098.598501:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432098.598513:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432098/real 1789432098] req@ffff9811d1bc4000 x1876350913592832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432098.598524:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913592832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432098.598533:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432098.598536:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432098.598540:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432098.598545:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432098.598560:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432098.598569:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432098/real 1789432098] req@ffff9811d1bc5880 x1876350913592960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432098.598586:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913592960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432098.598593:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432098.598596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432098.598599:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913592960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432098.598604:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432098.598618:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432098.598625:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432098/real 1789432098] req@ffff9811d1bc5f80 x1876350913593088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432098.598642:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913593088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432098.598651:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432098.598654:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432098.598656:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432099.494029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432100.518030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432101.542044:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432102.565979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432103.590013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432103.717941:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432103.717948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432103.717951:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432103.717959:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4473 00000100:00080000:0.0:1789432103.717965:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432103.717973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432103.717999:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913593216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432103.718016:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432103.718019:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432103.718020:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432103.718023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4473 00000100:00080000:0.0:1789432103.718025:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432103.718027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432103.718035:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432103.718044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913593344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432103.718044:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913593216:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432103.718056:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432103.718058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432103.718059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432103.718061:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4473 00000100:00080000:0.0:1789432103.718063:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432103.718065:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432103.718069:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432103.718076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913593472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432103.718082:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432103.718085:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432103.718086:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432103.718088:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4473 00000100:00080000:0.0:1789432103.718090:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432103.718092:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:2.0:1789432103.718095:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789432103.718096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432103.718106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913593600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432103.718127:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432103/real 1789432103] req@ffff9811c8cd5500 x1876350913593216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432103.718138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913593216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432103.718146:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432103.718149:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432103.718151:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913593216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432103.718167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432103.718179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432103.718186:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432103/real 1789432103] req@ffff9811c8cd6d80 x1876350913593344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432103.718195:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913593344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432103.718203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432103.718206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432103.718209:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432103.718214:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432103.718232:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432103.718243:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432103/real 1789432103] req@ffff9811c8cd5f80 x1876350913593472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432103.718250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913593472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432103.718255:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432103.718257:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432103.718259:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432103.718263:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593600:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432103.718276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432103.718294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432103/real 1789432103] req@ffff9811c8cd4a80 x1876350913593600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432103.718301:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913593600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432103.718307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432103.718309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432103.718310:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432104.613964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432105.637967:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432106.661968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432107.685999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432108.709933:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432108.837905:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432108.837915:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432108.837919:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432108.837928:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4478 00000100:00080000:0.0:1789432108.837937:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432108.837947:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432108.837981:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913593728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432108.838006:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432108.838010:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432108.838013:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432108.838018:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4478 00000100:00080000:0.0:1789432108.838022:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432108.838030:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432108.838031:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913593728:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789432108.838041:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432108.838057:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913593856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432108.838068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432108.838072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432108.838074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432108.838077:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4478 00000800:00000100:2.0:1789432108.838081:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432108.838084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432108.838100:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432108.838105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432108.838110:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432108/real 1789432108] req@ffff9811c8cd6a00 x1876350913593728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432108.838120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913593984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432108.838125:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913593728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432108.838143:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432108.838144:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432108.838148:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:0.0:1789432108.838152:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1789432108.838153:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913593728:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432108.838157:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432108.838163:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4478 00000100:00080000:0.0:1789432108.838167:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432108.838172:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593856:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789432108.838174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432108.838186:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432108.838196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432108.838204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432108/real 1789432108] req@ffff9811c8cd7b80 x1876350913593856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432108.838212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913594112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432108.838215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913593856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432108.838240:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432108.838255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432108.838259:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432108.838264:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432108.838279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432108.838286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432108/real 1789432108] req@ffff9811c8cd6300 x1876350913593984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432108.838293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913593984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432108.838297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432108.838299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432108.838301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913593984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432108.838309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432108.838318:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432108.838323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432108/real 1789432108] req@ffff9811c8cd7480 x1876350913594112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432108.838334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913594112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432108.838339:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432108.838340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432108.838342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432109.733973:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432110.757965:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432111.781984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432112.805974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432113.830055:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432113.957939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432113.957948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432113.957953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432113.957960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4483 00000100:00080000:1.0:1789432113.957966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432113.957975:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432113.958003:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913594240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432113.958023:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432113.958025:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432113.958026:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432113.958029:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4483 00000100:00080000:1.0:1789432113.958032:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432113.958035:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432113.958042:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432113.958047:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913594240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432113.958053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913594368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432113.958061:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432113.958064:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432113.958065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432113.958067:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4483 00000100:00080000:1.0:1789432113.958069:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432113.958071:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432113.958074:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432113.958081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913594496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432113.958087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432113.958089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432113.958090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432113.958092:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4483 00000100:00080000:1.0:1789432113.958095:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432113.958097:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432113.958100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432113.958105:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432113.958109:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913594624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432113.958141:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432113/real 1789432113] req@ffff9811d0bdfb80 x1876350913594240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432113.958154:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913594240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432113.958161:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432113.958164:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432113.958166:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913594240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432113.958186:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432113.958217:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432113.958226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432113/real 1789432113] req@ffff9811d0bdea00 x1876350913594368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432113.958235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913594368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432113.958241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432113.958244:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432113.958247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432113.958252:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432113.958270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432113.958280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432113/real 1789432113] req@ffff9811d0bde680 x1876350913594496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432113.958289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913594496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432113.958294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432113.958296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432113.958297:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432113.958301:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594624:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432113.958313:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432113.958319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432113/real 1789432113] req@ffff9811d0bdd180 x1876350913594624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432113.958325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913594624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432113.958334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432113.958336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432113.958337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432114.853994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432115.877978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432116.902012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432117.925971:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432118.949999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432119.077916:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432119.077923:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432119.077926:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432119.077934:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4488 00000100:00080000:1.0:1789432119.077940:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432119.077949:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432119.077977:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913594752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432119.077998:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432119.078001:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432119.078002:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432119.078005:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4488 00000100:00080000:1.0:1789432119.078007:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432119.078010:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432119.078017:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432119.078025:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913594752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432119.078026:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913594880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432119.078036:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432119.078038:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432119.078039:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432119.078041:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4488 00000100:00080000:1.0:1789432119.078042:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432119.078044:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432119.078047:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432119.078054:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913595008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432119.078060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432119.078061:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432119.078062:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432119.078065:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4488 00000100:00080000:1.0:1789432119.078068:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432119.078071:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432119.078074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432119.078082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913595136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432119.078085:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432119.078118:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432119/real 1789432119] req@ffff9811d0bdd500 x1876350913594752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432119.078131:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913594752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432119.078137:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432119.078140:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432119.078142:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913594752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432119.078163:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432119.078191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432119.078200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432119/real 1789432119] req@ffff9811d0bddc00 x1876350913594880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432119.078207:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913594880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432119.078212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432119.078214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432119.078215:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913594880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432119.078219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432119.078228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432119.078233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432119/real 1789432119] req@ffff9811d0bdc380 x1876350913595008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432119.078241:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913595008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432119.078247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432119.078250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432119.078253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432119.078258:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432119.078274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432119.078285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432119/real 1789432119] req@ffff9811d0bdd180 x1876350913595136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432119.078296:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913595136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432119.078303:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432119.078304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432119.078307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432119.974014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432120.998065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432122.021965:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432123.046040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432124.069989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432124.198001:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432124.198008:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432124.198012:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432124.198019:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4493 00000100:00080000:1.0:1789432124.198026:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432124.198033:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432124.198060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913595264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432124.198111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913595264:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789432124.198172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432124.198201:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432124.198205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432124.198207:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432124.198211:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4493 00000100:00080000:1.0:1789432124.198215:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432124.198220:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432124.198228:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432124.198241:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913595392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432124.198284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432124/real 1789432124] req@ffff9811d0bde680 x1876350913595264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432124.198297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913595264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432124.198303:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432124.198306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432124.198308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913595264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432124.198324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432124.198338:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432124.198346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432124/real 1789432124] req@ffff9811d0bdea00 x1876350913595392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432124.198352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913595392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432124.198357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432124.198358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432124.198361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595392:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789432124.198393:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432124.198397:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432124.198398:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432124.198401:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4493 00000100:00080000:1.0:1789432124.198404:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432124.198410:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432124.198413:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432124.198422:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913595520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432124.198431:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432124.198433:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432124.198435:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789432124.198437:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595520:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789432124.198439:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4493 00000100:00080000:1.0:1789432124.198444:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432124.198449:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432124.198453:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432124.198465:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913595648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432124.198470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432124.198487:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432124/real 1789432124] req@ffff9811d0bdfb80 x1876350913595520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432124.198498:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913595520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432124.198504:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432124.198506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432124.198509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432124.198518:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432124.198532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432124.198538:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432124/real 1789432124] req@ffff9811d0bdf480 x1876350913595648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432124.198545:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913595648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432124.198550:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432124.198551:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432124.198553:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432125.093972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432126.117983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432127.142002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432128.165948:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432129.190028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432129.317927:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432129.317934:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432129.317937:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432129.317945:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4498 00000100:00080000:3.0:1789432129.317951:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432129.317959:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432129.317985:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913595776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432129.318008:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432129.318010:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432129.318011:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432129.318014:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4498 00000100:00080000:3.0:1789432129.318020:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432129.318026:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432129.318033:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432129.318041:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913595776:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432129.318042:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913595904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432129.318055:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432129.318058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432129.318059:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432129.318062:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4498 00000100:00080000:3.0:1789432129.318065:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432129.318067:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432129.318070:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432129.318076:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913596032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432129.318082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432129.318085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432129.318086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432129.318088:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4498 00000100:00080000:3.0:1789432129.318090:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432129.318093:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432129.318106:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432129.318114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913596160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432129.318135:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432129.318164:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432129.318187:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432130.213951:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432131.238004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432132.261955:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432132.390209:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432132.390275:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432132.390296:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432132.390347:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432132.390360:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432132.390371:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789432132.390405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432129/real 1789432132] req@ffff9811d1bc6300 x1876350913595904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432132.390449:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913595904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432132.390462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432132.390481:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432132.390486:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913595904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432132.390495:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432129/real 1789432132] req@ffff9811d1bc5500 x1876350913596032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432132.390508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913596032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432132.390516:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432132.390521:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432132.390525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432132.390532:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432129/real 1789432132] req@ffff9811d1bc4700 x1876350913596160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432132.390543:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913596160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432132.390552:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432132.390556:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432132.390559:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596160:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789432132.390566:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432129/real 1789432132] req@ffff9811d1bc7100 x1876350913595776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432132.390579:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913595776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432132.390587:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432132.390591:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432132.390594:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913595776:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432133.286005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432134.310008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432134.437942:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432134.437949:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432134.437952:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432134.437959:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4504 00000100:00080000:3.0:1789432134.437967:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432134.437975:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432134.438001:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913596288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432134.438020:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432134.438023:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432134.438024:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432134.438026:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4504 00000100:00080000:3.0:1789432134.438028:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432134.438030:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432134.438037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432134.438046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913596416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432134.438054:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432134.438068:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432134.438070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432134.438073:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4504 00000100:00080000:3.0:1789432134.438076:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432134.438081:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432134.438085:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432134.438096:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913596544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432134.438105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432134.438108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432134.438110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432134.438114:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4504 00000100:00080000:3.0:1789432134.438117:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432134.438122:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432134.438126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432134.438142:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913596672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432134.438266:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913596288:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789432134.438332:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432134.438369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432134/real 1789432134] req@ffff9811d1bc6680 x1876350913596288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432134.438390:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913596288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432134.438400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432134.438404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432134.438407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913596288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432134.438414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432134.438437:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432134.438448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432134/real 1789432134] req@ffff9811d1bc7100 x1876350913596416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432134.438459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913596416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432134.438466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432134.438469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432134.438471:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432134.438477:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432134.438495:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432134.438505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432134/real 1789432134] req@ffff9811d1bc4700 x1876350913596544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432134.438515:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913596544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432134.438526:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432134.438528:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432134.438530:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432134.438535:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432134.438552:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432134.438561:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432134/real 1789432134] req@ffff9811d1bc5500 x1876350913596672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432134.438571:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913596672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432134.438578:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432134.438581:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432134.438583:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432135.333998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432136.358000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432137.382021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432138.406002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432139.429990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432139.557966:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432139.557976:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432139.557980:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432139.557988:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4509 00000100:00080000:0.0:1789432139.557997:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432139.558008:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432139.558054:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913596800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432139.558081:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432139.558084:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432139.558086:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432139.558090:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4509 00000100:00080000:0.0:1789432139.558093:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432139.558098:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432139.558106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432139.558120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913596928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432139.558130:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913596800:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432139.558133:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432139.558138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432139.558139:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432139.558143:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4509 00000100:00080000:0.0:1789432139.558147:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432139.558152:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432139.558158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432139.558171:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913597056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432139.558181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789432139.558183:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432139.558187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432139.558190:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432139.558204:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4509 00000100:00080000:0.0:1789432139.558208:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432139.558213:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432139.558218:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432139.558220:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432139/real 1789432139] req@ffff9811c8cd5180 x1876350913596800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432139.558234:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913597184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432139.558238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913596800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432139.558262:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432139.558266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432139.558270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913596800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432139.558288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432139.558303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432139.558311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432139/real 1789432139] req@ffff9811c8cd7800 x1876350913596928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432139.558318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913596928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432139.558324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432139.558326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432139.558328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913596928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432139.558331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432139.558340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432139.558345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432139/real 1789432139] req@ffff9811c8cd4000 x1876350913597056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432139.558350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913597056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432139.558355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432139.558356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432139.558358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432139.558361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597184:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432139.558370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432139.558374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432139/real 1789432139] req@ffff9811c8cd7100 x1876350913597184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432139.558380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913597184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432139.558385:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432139.558386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432139.558388:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432140.453979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432141.478003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432142.501994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432143.526001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432144.550018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432144.677938:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432144.677946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432144.677949:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432144.677957:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4514 00000100:00080000:1.0:1789432144.677964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432144.677972:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432144.678013:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913597312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432144.678032:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432144.678035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432144.678036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432144.678039:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4514 00000100:00080000:1.0:1789432144.678041:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432144.678044:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432144.678051:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432144.678058:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913597440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432144.678064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432144.678067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432144.678068:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432144.678070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4514 00000100:00080000:1.0:1789432144.678072:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432144.678074:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432144.678077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432144.678085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913597568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432144.678086:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913597312:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432144.678096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432144.678099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432144.678100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432144.678102:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4514 00000100:00080000:1.0:1789432144.678104:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432144.678106:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432144.678110:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432144.678116:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913597696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432144.678149:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432144.678195:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432144/real 1789432144] req@ffff9811d0bded80 x1876350913597312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432144.678214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913597312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432144.678224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432144.678229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432144.678232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913597312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432144.678239:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432144.678261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432144.678273:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432144/real 1789432144] req@ffff9811d0bdc000 x1876350913597440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432144.678285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913597440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432144.678294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432144.678297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432144.678300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432144.678323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432144.678341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432144.678351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432144/real 1789432144] req@ffff9811d0bdc700 x1876350913597568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432144.678361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913597568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432144.678369:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432144.678372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432144.678375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432144.678381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432144.678399:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432144.678410:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432144/real 1789432144] req@ffff9811d0bdd880 x1876350913597696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432144.678421:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913597696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432144.678438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432144.678444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432144.678447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432145.573959:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432146.597992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432147.621991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432148.646026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432149.669966:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432149.797937:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432149.797957:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432149.797961:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432149.797968:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4519 00000100:00080000:3.0:1789432149.797974:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432149.797982:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432149.798008:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913597824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432149.798026:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432149.798028:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432149.798029:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432149.798032:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4519 00000100:00080000:3.0:1789432149.798034:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432149.798036:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432149.798042:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432149.798049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913597952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432149.798055:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432149.798059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432149.798060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432149.798063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4519 00000100:00080000:3.0:1789432149.798065:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432149.798067:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432149.798070:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432149.798075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913597824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432149.798078:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913598080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432149.798088:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432149.798090:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432149.798091:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432149.798093:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4519 00000100:00080000:3.0:1789432149.798095:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432149.798097:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432149.798100:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432149.798106:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913598208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432149.798138:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432149.798180:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432149/real 1789432149] req@ffff9811d1bc6300 x1876350913597824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432149.798197:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913597824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432149.798207:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432149.798211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432149.798216:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913597824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432149.798223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432149.798243:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432149.798255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432149/real 1789432149] req@ffff9811d1bc5f80 x1876350913597952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432149.798267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913597952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432149.798274:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432149.798277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432149.798280:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913597952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432149.798298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432149.798315:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432149.798325:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432149/real 1789432149] req@ffff9811d1bc5880 x1876350913598080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432149.798335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913598080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432149.798342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432149.798345:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432149.798348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432149.798353:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432149.798369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432149.798378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432149/real 1789432149] req@ffff9811d1bc4000 x1876350913598208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432149.798387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913598208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432149.798400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432149.798403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432149.798405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432150.694032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432151.718082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432152.742008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432153.765987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432154.790016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432154.917960:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432154.917969:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432154.917973:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432154.917982:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4524 00000100:00080000:0.0:1789432154.917989:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432154.918007:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432154.918044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913598336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432154.918068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432154.918072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432154.918074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432154.918078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4524 00000100:00080000:0.0:1789432154.918081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432154.918086:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432154.918094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432154.918106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913598464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432154.918115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432154.918119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432154.918121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432154.918125:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4524 00000100:00080000:0.0:1789432154.918128:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432154.918133:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432154.918138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432154.918142:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913598336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432154.918151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913598592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432154.918169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432154.918173:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432154.918175:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432154.918179:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4524 00000100:00080000:0.0:1789432154.918182:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432154.918186:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432154.918191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432154.918196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432154.918206:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913598720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432154.918233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432154/real 1789432154] req@ffff9811c8cd5c00 x1876350913598336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432154.918250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913598336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432154.918261:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432154.918264:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432154.918268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913598336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432154.918274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432154.918290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432154.918299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432154/real 1789432154] req@ffff9811c8cd4700 x1876350913598464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432154.918310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913598464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432154.918318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432154.918321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432154.918324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432154.918342:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432154.918354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432154.918362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432154/real 1789432154] req@ffff9811c8cd6680 x1876350913598592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432154.918372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913598592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432154.918378:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432154.918381:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432154.918384:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432154.918388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598720:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432154.918401:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432154.918408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432154/real 1789432154] req@ffff9811c8cd7100 x1876350913598720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432154.918418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913598720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432154.918425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432154.918428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432154.918431:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432155.813982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432156.838020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432157.862014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432158.886008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432159.910004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432160.037975:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432160.037985:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432160.037990:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432160.037999:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4529 00000100:00080000:1.0:1789432160.038008:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432160.038019:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432160.038051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913598848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432160.038077:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432160.038080:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432160.038082:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432160.038086:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4529 00000100:00080000:1.0:1789432160.038089:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432160.038093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432160.038101:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432160.038111:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913598976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432160.038121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432160.038126:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432160.038128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432160.038134:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4529 00000100:00100000:2.0:1789432160.038136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913598848:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432160.038138:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432160.038144:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432160.038149:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432160.038158:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913599104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432160.038172:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432160.038175:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432160.038177:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432160.038181:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4529 00000100:00080000:1.0:1789432160.038184:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432160.038188:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432160.038193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432160.038198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432160.038207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913599232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432160.038239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432160/real 1789432160] req@ffff9811d0bde300 x1876350913598848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432160.038259:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913598848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432160.038270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432160.038274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432160.038277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913598848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432160.038284:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432160.038306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432160.038318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432160/real 1789432160] req@ffff9811d0bdf100 x1876350913598976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432160.038329:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913598976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432160.038336:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432160.038339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432160.038343:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913598976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432160.038363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432160.038378:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432160.038386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432160/real 1789432160] req@ffff9811d0bdf800 x1876350913599104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432160.038396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913599104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432160.038403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432160.038405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432160.038408:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432160.038413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432160.038426:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432160.038435:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432160/real 1789432160] req@ffff9811d0bdd880 x1876350913599232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432160.038452:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913599232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432160.038462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432160.038465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432160.038469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432160.934012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432161.957985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432162.982069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432164.006055:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432165.030090:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432165.157973:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432165.157980:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432165.157984:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432165.157992:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4534 00000100:00080000:3.0:1789432165.158000:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432165.158011:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432165.158042:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913599360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432165.158065:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432165.158068:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432165.158070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432165.158074:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4534 00000100:00080000:3.0:1789432165.158076:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432165.158081:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432165.158090:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432165.158112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913599488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432165.158118:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913599360:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432165.158127:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432165.158131:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432165.158132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432165.158137:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4534 00000100:00080000:3.0:1789432165.158141:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432165.158146:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432165.158157:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432165.158169:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913599616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432165.158175:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432165.158181:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432165.158184:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432165.158186:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432165.158189:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4534 00000100:00080000:3.0:1789432165.158193:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432165.158199:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432165.158202:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432165.158209:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913599744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432165.158220:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432165/real 1789432165] req@ffff9811d1bc7480 x1876350913599360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432165.158239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913599360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432165.158249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432165.158252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432165.158254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913599360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432165.158271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432165.158287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432165.158295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432165/real 1789432165] req@ffff9811d1bc5c00 x1876350913599488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432165.158303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913599488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432165.158308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432165.158310:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432165.158312:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432165.158315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432165.158326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432165.158331:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432165/real 1789432165] req@ffff9811d1bc7800 x1876350913599616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432165.158337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913599616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432165.158341:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432165.158343:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432165.158345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432165.158347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432165.158357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432165.158361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432165/real 1789432165] req@ffff9811d1bc5180 x1876350913599744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432165.158367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913599744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432165.158372:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432165.158374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432165.158375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913599744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432166.054080:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432167.077988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432168.102004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432169.125940:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432170.149943:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432170.277912:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432170.277920:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432170.277923:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432170.277931:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4539 00000100:00080000:3.0:1789432170.277938:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432170.277948:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432170.277976:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913599872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432170.277997:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432170.278001:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432170.278002:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432170.278006:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4539 00000100:00080000:3.0:1789432170.278010:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432170.278016:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432170.278025:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432170.278041:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913599872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432170.278041:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913600000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432170.278057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432170.278062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432170.278064:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432170.278067:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4539 00000100:00080000:3.0:1789432170.278071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432170.278075:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432170.278080:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432170.278090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913600128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432170.278099:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432170.278102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432170.278114:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432170.278116:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432170.278120:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4539 00000100:00080000:3.0:1789432170.278125:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432170.278131:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432170/real 1789432170] req@ffff9811d1bc6a00 x1876350913599872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432170.278135:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432170.278144:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432170.278150:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913599872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432170.278158:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789432170.278158:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913600256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432170.278165:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432170.278171:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913599872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432170.278191:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432170.278206:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432170.278213:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432170/real 1789432170] req@ffff9811d1bc5180 x1876350913600000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432170.278220:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913600000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432170.278224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432170.278226:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432170.278228:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432170.278231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432170.278239:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432170.278244:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432170/real 1789432170] req@ffff9811d1bc7800 x1876350913600128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432170.278254:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913600128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432170.278258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432170.278259:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432170.278261:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432170.278264:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600256:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432170.278273:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432170.278277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432170/real 1789432170] req@ffff9811d1bc5c00 x1876350913600256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432170.278283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913600256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432170.278287:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432170.278289:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432170.278291:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432171.174004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432172.197984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432173.222000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432174.246000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432175.269976:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432175.397906:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432175.397917:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432175.397921:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432175.397930:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4544 00000100:00080000:3.0:1789432175.397945:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432175.397956:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432175.397990:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913600384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432175.398014:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432175.398018:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432175.398020:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432175.398024:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4544 00000100:00080000:3.0:1789432175.398028:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432175.398033:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432175.398043:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432175.398051:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913600384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432175.398060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913600512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432175.398071:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432175.398074:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432175.398076:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432175.398079:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4544 00000100:00080000:3.0:1789432175.398083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432175.398102:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432175.398107:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432175.398110:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432175.398124:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913600640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432175.398135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432175.398138:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432175.398140:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432175.398144:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4544 00000100:00080000:3.0:1789432175.398149:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432175.398150:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432175/real 1789432175] req@ffff9811d1bc7480 x1876350913600384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432175.398156:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432175.398162:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432175.398179:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913600384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432175.398188:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913600768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432175.398192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432175.398202:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432175.398208:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913600384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432175.398230:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432175.398250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432175.398262:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432175/real 1789432175] req@ffff9811d1bc4000 x1876350913600512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432175.398272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913600512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432175.398279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432175.398282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432175.398285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432175.398289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432175.398304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432175.398314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432175/real 1789432175] req@ffff9811d1bc5880 x1876350913600640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432175.398325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913600640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432175.398337:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432175.398339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432175.398342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432175.398347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432175.398364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432175.398373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432175/real 1789432175] req@ffff9811d1bc5f80 x1876350913600768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432175.398382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913600768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432175.398390:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432175.398392:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432175.398395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913600768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432176.293937:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432177.317982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432178.342009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432179.365985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432180.389989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432180.517912:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432180.517921:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432180.517924:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432180.517932:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4550 00000100:00080000:3.0:1789432180.517938:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432180.517946:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432180.517984:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913600896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432180.518010:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432180.518015:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432180.518016:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432180.518019:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4550 00000100:00080000:3.0:1789432180.518022:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432180.518025:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432180.518033:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432180.518048:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913600896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432180.518049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913601024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432180.518061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432180.518065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432180.518067:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432180.518069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4550 00000100:00080000:3.0:1789432180.518071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432180.518073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432180.518076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432180.518083:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913601152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432180.518090:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432180.518092:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432180.518093:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432180.518096:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4550 00000100:00080000:3.0:1789432180.518098:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432180.518100:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432180.518102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432180.518109:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432180.518110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913601280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432180.518144:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432180/real 1789432180] req@ffff9811d1bc6300 x1876350913600896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432180.518156:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913600896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432180.518163:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432180.518166:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432180.518168:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913600896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432180.518184:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432180.518198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432180.518205:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432180/real 1789432180] req@ffff9811d1bc5500 x1876350913601024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432180.518211:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913601024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432180.518216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432180.518218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432180.518220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432180.518223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432180.518231:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432180.518236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432180/real 1789432180] req@ffff9811d1bc4700 x1876350913601152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432180.518242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913601152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432180.518247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432180.518248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432180.518250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432180.518253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601280:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432180.518262:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432180.518267:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432180/real 1789432180] req@ffff9811d1bc7100 x1876350913601280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432180.518273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913601280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432180.518278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432180.518280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432180.518281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432181.413947:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432182.437986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432183.462015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789432184.485999:0:333760:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1789432184.486032:0:6161:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1789432184.486039:0:6161:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1789432184.486065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1789432184.486070:0:6161:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x6a9:0x0]: losing layout lock 00010000:00010000:2.0:1789432184.486089:0:6161:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789432184.486154:0:6161:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 1/0,0 mode: --/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789432184.486162:0:6161:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 1/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789432184.486173:0:6161:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1789432184.486202:0:6161:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811d0a63000/0x1ffb2ee1bd27ebc6 lrc: 0/0,0 mode: --/CR res: [0x200000402:0x6a9:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x4fe1f540795ad7cf expref: -99 pid: 333673 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1789432184.486221:0:6161:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff9811d1e45800 lock: ffff9811c121e800/0x1ffb2ee1bd27eba3 lrc: 0/0,0 mode: --/PR res: [0x200000402:0x6a7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x4fe1f540795ad766 expref: -99 pid: 333674 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1789432184.486234:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432184.486237:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432184.486237:0:2391:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff9811d2102300 x1876350913601408/t0(0) o103->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1789432184.486241:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432185.510003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432185.637935:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432185.637944:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432185.637948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432185.637956:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4555 00000100:00080000:1.0:1789432185.637965:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432185.637975:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432185.638011:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913601536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432185.638035:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432185.638038:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432185.638040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432185.638045:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4555 00000100:00080000:1.0:1789432185.638049:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432185.638053:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432185.638061:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913601536:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789432185.638063:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432185.638078:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913601664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432185.638088:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432185.638092:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432185.638093:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432185.638097:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4555 00000100:00080000:1.0:1789432185.638101:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432185.638105:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432185.638110:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789432185.638115:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432185.638122:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913601792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432185.638133:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432185.638136:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432185.638137:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432185.638142:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4555 00000100:00080000:1.0:1789432185.638146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432185.638150:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432185.638153:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432185/real 1789432185] req@ffff9811d0bdc700 x1876350913601536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789432185.638156:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432185.638170:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913601920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432185.638174:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913601536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432185.638192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432185.638195:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432185.638198:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913601536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432185.638214:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432185.638228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432185.638237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432185/real 1789432185] req@ffff9811d0bdc000 x1876350913601664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432185.638261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913601664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432185.638269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432185.638272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432185.638275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432185.638287:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432185.638303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432185.638311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432185/real 1789432185] req@ffff9811d0bded80 x1876350913601792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432185.638318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913601792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432185.638322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432185.638324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432185.638326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432185.638329:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432185.638339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432185.638344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432185/real 1789432185] req@ffff9811d0bdf480 x1876350913601920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432185.638355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913601920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432185.638364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432185.638366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432185.638367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913601920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432186.534002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432187.558074:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432188.581974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432189.605990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432190.629958:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432190.757953:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432190.757961:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432190.757965:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432190.757973:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4560 00000100:00080000:2.0:1789432190.757980:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432190.757989:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432190.758020:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913602048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432190.758049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432190.758053:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432190.758056:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432190.758061:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4560 00000100:00080000:2.0:1789432190.758066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432190.758073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432190.758080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913602048:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789432190.758084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432190.758102:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913602176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432190.758122:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432190.758126:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432190.758128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432190.758132:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4560 00000800:00000100:1.0:1789432190.758135:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432190.758148:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432190.758156:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432190.758162:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432190.758172:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432190/real 1789432190] req@ffff9811d2103b80 x1876350913602048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432190.758179:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913602304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432190.758191:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913602048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432190.758194:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432190.758205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432190.758208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432190.758208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432190.758220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:2.0:1789432190.758220:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4560 00000100:00100000:1.0:1789432190.758226:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913602048:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432190.758229:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432190.758237:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432190.758243:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432190.758247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432190.758257:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913602432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432190.758267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432190.758281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432190/real 1789432190] req@ffff9811d2101c00 x1876350913602176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432190.758290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913602176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432190.758297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432190.758300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432190.758303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432190.758307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432190.758317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432190.758322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432190/real 1789432190] req@ffff9811d2102a00 x1876350913602304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432190.758330:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913602304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432190.758335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432190.758336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432190.758338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432190.758346:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432190.758356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432190.758361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432190/real 1789432190] req@ffff9811d2101f80 x1876350913602432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432190.758368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913602432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432190.758374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432190.758376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432190.758377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432191.654033:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432192.678019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432193.701999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432194.726038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432195.750011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432195.877962:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432195.877973:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432195.877979:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432195.877990:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4565 00000100:00080000:2.0:1789432195.878000:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432195.878012:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432195.878049:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913602560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432195.878076:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432195.878081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432195.878083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432195.878089:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4565 00000100:00080000:2.0:1789432195.878094:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432195.878101:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432195.878112:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432195.878113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913602560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432195.878130:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913602688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432195.878144:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432195.878148:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432195.878150:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432195.878155:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4565 00000100:00080000:2.0:1789432195.878159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432195.878175:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432195.878181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432195.878197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913602816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432195.878211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602688:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789432195.878224:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432195.878230:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432195.878232:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432195.878238:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4565 00000100:00080000:2.0:1789432195.878244:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432195.878250:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432195.878256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602816:192.168.203.128@tcp:38:kworker.0 00000008:00100000:2.0:1789432195.878257:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432195.878276:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913602944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432195.878306:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432196.774033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432197.798006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432198.822053:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432198.950126:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432198.950180:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432198.950196:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432198.950233:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432198.950244:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432198.950252:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432198.950274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432195/real 1789432198] req@ffff9811d2101f80 x1876350913602944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432198.950306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913602944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432198.950328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432198.950334:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432198.950344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602944:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432198.950369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432195/real 1789432198] req@ffff9811d2101180 x1876350913602816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432198.950381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913602816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432198.950388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432198.950391:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432198.950394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432198.950399:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432195/real 1789432198] req@ffff9811d2101880 x1876350913602688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432198.950407:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913602688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432198.950414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432198.950433:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432198.950435:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913602688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432198.950441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432195/real 1789432198] req@ffff9811d2100e00 x1876350913602560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432198.950469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913602560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432198.950479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432198.950484:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432198.950487:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913602560:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432199.846023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432200.870045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432200.997982:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432200.997995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432200.997999:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432200.998012:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4570 00000100:00080000:1.0:1789432200.998023:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432200.998037:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432200.998078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913603072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432200.998111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432200.998115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432200.998118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432200.998123:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4570 00000100:00080000:1.0:1789432200.998128:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432200.998136:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432200.998149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432200.998165:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913603072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432200.998167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913603200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432200.998182:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432200.998197:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432200.998199:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432200.998208:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4570 00000100:00080000:1.0:1789432200.998213:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432200.998219:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432200.998225:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432200.998238:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913603328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432200.998250:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432200.998255:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1789432200.998258:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432200.998259:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432200.998268:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4570 00000100:00080000:1.0:1789432200.998273:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432200.998278:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432200.998285:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432200.998298:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913603456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432200.998324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432200/real 1789432200] req@ffff9811d0bdfb80 x1876350913603072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432200.998349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913603072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432200.998363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432200.998367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432200.998373:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913603072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432200.998406:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432200.998436:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432200.998452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432200/real 1789432200] req@ffff9811d0bdea00 x1876350913603200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432200.998480:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913603200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432200.998490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432200.998494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432200.998498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432200.998506:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432200.998530:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432200.998544:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432200/real 1789432200] req@ffff9811d0bde680 x1876350913603328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432200.998557:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913603328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432200.998566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432200.998570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432200.998573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432200.998591:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432200.998614:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432200.998627:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432200/real 1789432200] req@ffff9811d0bdd180 x1876350913603456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432200.998646:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913603456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432200.998654:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432200.998657:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432200.998661:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432201.894037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432202.918060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432203.941980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432204.966001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432205.989945:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432206.117991:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432206.118002:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432206.118007:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432206.118017:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4575 00000100:00080000:3.0:1789432206.118028:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432206.118040:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432206.118084:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913603584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432206.118111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432206.118114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432206.118116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432206.118119:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4575 00000100:00080000:3.0:1789432206.118122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432206.118125:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432206.118136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432206.118149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913603712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432206.118154:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913603584:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432206.118159:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432206.118163:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432206.118165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432206.118169:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4575 00000100:00080000:3.0:1789432206.118171:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432206.118173:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432206.118176:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432206.118185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913603840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432206.118191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432206.118195:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432206.118196:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432206.118199:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4575 00000100:00080000:3.0:1789432206.118220:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432206.118223:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432206.118226:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432206.118232:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432206.118236:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913603968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432206.118276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432206/real 1789432206] req@ffff9811d1bc6680 x1876350913603584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432206.118291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913603584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432206.118301:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432206.118304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432206.118307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913603584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432206.118328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432206.118344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432206.118352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432206/real 1789432206] req@ffff9811d1bc7100 x1876350913603712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432206.118365:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913603712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432206.118371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432206.118372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432206.118374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432206.118378:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432206.118388:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432206.118394:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432206/real 1789432206] req@ffff9811d1bc4700 x1876350913603840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432206.118402:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913603840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432206.118407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432206.118409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432206.118410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432206.118414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603968:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432206.118424:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432206.118430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432206/real 1789432206] req@ffff9811d1bc5500 x1876350913603968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432206.118436:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913603968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432206.118446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432206.118449:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432206.118451:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913603968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432207.014009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432208.037933:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432209.062019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432210.085989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432211.110026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432211.237918:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432211.237927:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432211.237930:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432211.237940:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4580 00000100:00080000:3.0:1789432211.237947:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432211.237957:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432211.237989:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913604096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432211.238008:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432211.238011:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432211.238013:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432211.238016:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4580 00000100:00080000:3.0:1789432211.238018:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432211.238022:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432211.238030:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432211.238039:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913604096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432211.238041:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913604224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432211.238053:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432211.238056:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432211.238058:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432211.238060:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4580 00000100:00080000:3.0:1789432211.238063:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432211.238066:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432211.238069:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432211.238078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913604352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432211.238085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432211.238088:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432211.238089:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432211.238091:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4580 00000100:00080000:3.0:1789432211.238094:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789432211.238097:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432211.238101:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432211.238105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432211.238115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913604480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432211.238133:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432211/real 1789432211] req@ffff9811d1bc6300 x1876350913604096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432211.238148:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913604096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432211.238156:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432211.238159:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432211.238162:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913604096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432211.238178:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432211.238192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432211.238200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432211/real 1789432211] req@ffff9811d1bc5f80 x1876350913604224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432211.238208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913604224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432211.238218:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432211.238220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432211.238222:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432211.238225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432211.238235:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432211.238241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432211/real 1789432211] req@ffff9811d1bc5880 x1876350913604352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432211.238248:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913604352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432211.238253:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432211.238255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432211.238257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432211.238260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432211.238282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432211.238289:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432211/real 1789432211] req@ffff9811d1bc4000 x1876350913604480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432211.238297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913604480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432211.238302:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432211.238304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432211.238307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432212.133998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432213.158024:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432214.181979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432215.206018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432216.229982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432216.357935:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432216.357947:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432216.357952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432216.357963:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4585 00000100:00080000:3.0:1789432216.357972:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432216.357984:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432216.358021:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913604608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432216.358051:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432216.358056:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432216.358058:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432216.358062:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4585 00000100:00080000:3.0:1789432216.358067:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432216.358073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432216.358083:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432216.358100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913604736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432216.358104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913604608:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432216.358122:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432216.358126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432216.358128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432216.358133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4585 00000100:00080000:3.0:1789432216.358137:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432216.358143:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432216.358149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432216.358162:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913604864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432216.358175:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432216.358177:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432216.358182:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432216.358185:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432216.358190:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4585 00000100:00080000:3.0:1789432216.358197:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432216.358203:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432216.358209:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432216.358216:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432216/real 1789432216] req@ffff9811d1bc7480 x1876350913604608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432216.358226:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913604992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432216.358235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913604608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432216.358251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432216.358255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432216.358258:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913604608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432216.358278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432216.358300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432216.358313:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432216/real 1789432216] req@ffff9811d1bc5c00 x1876350913604736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432216.358336:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913604736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432216.358344:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432216.358347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432216.358349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432216.358354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432216.358370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432216.358378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432216/real 1789432216] req@ffff9811d1bc7800 x1876350913604864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432216.358386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913604864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432216.358392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432216.358394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432216.358396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432216.358400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432216.358411:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432216.358416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432216/real 1789432216] req@ffff9811d1bc5180 x1876350913604992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432216.358424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913604992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432216.358432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432216.358436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432216.358439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913604992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432217.254014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432218.277937:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432219.301985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432220.325996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432221.349983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432221.477948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432221.477957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432221.477960:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432221.477967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4591 00000100:00080000:0.0:1789432221.477974:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432221.477984:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432221.478015:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913605120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432221.478036:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432221.478039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432221.478040:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432221.478043:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4591 00000100:00080000:0.0:1789432221.478045:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432221.478048:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432221.478056:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432221.478065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913605248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432221.478073:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432221.478079:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1789432221.478079:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913605120:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432221.478081:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432221.478088:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4591 00000100:00080000:0.0:1789432221.478090:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432221.478094:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432221.478098:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432221.478106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913605376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432221.478113:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432221.478116:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432221.478118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432221.478120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4591 00000100:00080000:0.0:1789432221.478123:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432221.478125:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432221.478129:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432221.478136:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913605504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432221.478143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432221.478196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432221/real 1789432221] req@ffff9811c8cd4000 x1876350913605120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432221.478210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913605120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432221.478219:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432221.478222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432221.478224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913605120:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432221.478241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432221.478259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432221.478267:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432221/real 1789432221] req@ffff9811c8cd7800 x1876350913605248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432221.478276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913605248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432221.478281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432221.478283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432221.478286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432221.478289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432221.478300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432221.478305:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432221/real 1789432221] req@ffff9811c8cd5180 x1876350913605376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432221.478318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913605376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432221.478323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432221.478324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432221.478326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432221.478330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605504:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432221.478341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432221.478347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432221/real 1789432221] req@ffff9811c8cd7480 x1876350913605504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432221.478354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913605504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432221.478360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432221.478361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432221.478363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432222.373990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432223.397982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432224.421994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432225.445989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432226.469990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432226.597961:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432226.597969:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432226.597973:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432226.597982:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4596 00000100:00080000:1.0:1789432226.597990:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432226.597999:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432226.598030:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913605632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432226.598053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432226.598056:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432226.598058:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432226.598061:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4596 00000100:00080000:1.0:1789432226.598063:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432226.598066:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432226.598073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432226.598082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913605760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432226.598090:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432226.598093:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432226.598094:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432226.598097:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4596 00000100:00080000:1.0:1789432226.598099:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432226.598104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913605632:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432226.598105:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432226.598113:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432226.598122:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913605888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432226.598130:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432226.598132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432226.598134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432226.598137:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4596 00000100:00080000:1.0:1789432226.598140:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432226.598143:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432226.598146:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432226.598152:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913606016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432226.598162:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432226.598196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432226/real 1789432226] req@ffff9811d0bdc380 x1876350913605632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432226.598214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913605632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432226.598221:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432226.598224:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432226.598227:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913605632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432226.598232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432226.598246:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432226.598254:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432226/real 1789432226] req@ffff9811d0bddc00 x1876350913605760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432226.598261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913605760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432226.598269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432226.598272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432226.598276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432226.598298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432226.598319:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432226.598332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432226/real 1789432226] req@ffff9811d0bdd500 x1876350913605888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432226.598341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913605888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432226.598350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432226.598353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432226.598355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913605888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432226.598359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432226.598373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432226.598379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432226/real 1789432226] req@ffff9811d0bdd180 x1876350913606016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432226.598386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913606016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432226.598410:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432226.598412:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432226.598414:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432227.494000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432228.518004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432229.541992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432230.565973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432231.589968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432231.717950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432231.717959:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432231.717963:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432231.717971:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4601 00000100:00080000:2.0:1789432231.717979:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432231.717988:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432231.718022:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913606144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432231.718048:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432231.718052:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432231.718054:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432231.718058:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4601 00000100:00080000:2.0:1789432231.718062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432231.718068:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432231.718077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432231.718083:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913606144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432231.718093:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913606272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432231.718105:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432231.718107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432231.718108:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432231.718110:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4601 00000100:00080000:2.0:1789432231.718113:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432231.718117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432231.718121:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432231.718129:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913606400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432231.718137:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432231.718140:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432231.718142:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432231.718145:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432231.718149:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4601 00000100:00080000:2.0:1789432231.718154:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432231.718160:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432231.718165:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432231.718178:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913606528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432231.718189:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432231/real 1789432231] req@ffff9811d2102a00 x1876350913606144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432231.718209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913606144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432231.718220:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432231.718224:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432231.718227:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913606144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432231.718249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432231.718272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432231.718285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432231/real 1789432231] req@ffff9811d2101c00 x1876350913606272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432231.718297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913606272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432231.718305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432231.718308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432231.718311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432231.718316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432231.718334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432231.718344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432231/real 1789432231] req@ffff9811d2103b80 x1876350913606400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432231.718368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913606400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432231.718376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432231.718380:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432231.718383:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432231.718389:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606528:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432231.718407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432231.718418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432231/real 1789432231] req@ffff9811d2101500 x1876350913606528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432231.718436:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913606528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432231.718443:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432231.718446:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432231.718450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432232.613961:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432233.637984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432234.661994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432235.686009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432236.709966:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432236.837929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432236.837936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432236.837939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432236.837947:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4606 00000100:00080000:0.0:1789432236.837953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432236.837964:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432236.837993:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913606656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432236.838012:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432236.838015:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432236.838017:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432236.838023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4606 00000100:00080000:0.0:1789432236.838026:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432236.838032:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432236.838037:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913606656:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789432236.838041:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432236.838056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913606784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432236.838064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432236.838066:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432236.838067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432236.838070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4606 00000100:00080000:0.0:1789432236.838083:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432236.838087:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432236.838090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432236.838099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913606912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432236.838100:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432236.838109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432236.838111:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432236.838112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432236.838115:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4606 00000100:00080000:0.0:1789432236.838118:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432236.838121:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432236.838124:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432236.838132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913607040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432236.838138:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432236/real 1789432236] req@ffff9811c8cd6300 x1876350913606656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432236.838154:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913606656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432236.838173:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432236.838177:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432236.838181:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913606656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432236.838207:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432236.838230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432236.838239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432236/real 1789432236] req@ffff9811c8cd7b80 x1876350913606784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432236.838247:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913606784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432236.838256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432236.838258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432236.838260:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432236.838264:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432236.838275:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432236.838280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432236/real 1789432236] req@ffff9811c8cd6a00 x1876350913606912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432236.838286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913606912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432236.838291:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432236.838292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432236.838294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913606912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432236.838297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432236.838306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432236.838312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432236/real 1789432236] req@ffff9811c8cd4a80 x1876350913607040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432236.838331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913607040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432236.838336:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432236.838338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432236.838340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432237.734061:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432238.758012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432239.781998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432240.805981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432241.829981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432241.957914:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432241.957922:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432241.957925:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432241.957932:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4611 00000100:00080000:3.0:1789432241.957944:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432241.957953:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432241.957980:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913607168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432241.957997:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432241.958000:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432241.958001:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432241.958003:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4611 00000100:00080000:3.0:1789432241.958005:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432241.958008:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432241.958014:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432241.958023:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913607296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432241.958031:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432241.958033:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432241.958035:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432241.958039:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4611 00000100:00100000:1.0:1789432241.958040:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913607168:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432241.958041:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432241.958047:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432241.958051:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432241.958057:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913607424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432241.958068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432241.958071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432241.958072:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432241.958074:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4611 00000100:00080000:3.0:1789432241.958076:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432241.958079:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432241.958082:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432241.958089:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913607552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432241.958101:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432241.958145:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432241/real 1789432241] req@ffff9811d1bc6a00 x1876350913607168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432241.958162:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913607168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432241.958172:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432241.958176:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432241.958179:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913607168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432241.958197:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432241.958217:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432241.958229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432241/real 1789432241] req@ffff9811d1bc5180 x1876350913607296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432241.958240:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913607296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432241.958248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432241.958251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432241.958254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432241.958259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432241.958276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432241.958301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432241/real 1789432241] req@ffff9811d1bc7800 x1876350913607424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432241.958318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913607424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432241.958325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432241.958328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432241.958331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432241.958337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432241.958355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432241.958365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432241/real 1789432241] req@ffff9811d1bc5c00 x1876350913607552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432241.958376:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913607552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432241.958383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432241.958386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432241.958389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432242.853984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432243.877983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432244.902001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432245.925990:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432246.949981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432247.077948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432247.077957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432247.077961:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432247.077968:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4616 00000100:00080000:1.0:1789432247.077975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432247.077983:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432247.078011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913607680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432247.078031:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432247.078034:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432247.078035:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432247.078038:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4616 00000100:00080000:1.0:1789432247.078040:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432247.078043:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432247.078050:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432247.078059:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913607808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432247.078064:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913607680:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432247.078079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432247.078081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432247.078083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432247.078085:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4616 00000100:00080000:1.0:1789432247.078087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432247.078090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432247.078093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432247.078102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913607936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432247.078112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432247.078115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432247.078116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432247.078121:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4616 00000800:00000100:2.0:1789432247.078125:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432247.078127:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432247.078134:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432247.078140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432247.078153:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913608064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432247.078162:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432247/real 1789432247] req@ffff9811d0bde680 x1876350913607680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432247.078179:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913607680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432247.078189:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432247.078192:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432247.078195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913607680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432247.078217:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432247.078237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432247.078262:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432247/real 1789432247] req@ffff9811d0bdea00 x1876350913607808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432247.078273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913607808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432247.078280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432247.078283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432247.078286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432247.078291:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432247.078307:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432247.078317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432247/real 1789432247] req@ffff9811d0bdfb80 x1876350913607936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432247.078326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913607936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432247.078333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432247.078336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432247.078339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913607936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432247.078344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608064:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432247.078360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432247.078370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432247/real 1789432247] req@ffff9811d0bdf480 x1876350913608064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432247.078380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913608064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432247.078387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432247.078389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432247.078393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432247.974054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432248.997940:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432250.021966:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432251.045985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432252.069997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432252.197950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432252.197959:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432252.197962:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432252.197971:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4621 00000100:00080000:2.0:1789432252.197979:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432252.197989:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432252.198022:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913608192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432252.198049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432252.198052:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432252.198054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432252.198059:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4621 00000100:00080000:2.0:1789432252.198062:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432252.198068:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432252.198076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432252.198080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913608192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432252.198091:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913608320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432252.198102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432252.198105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432252.198107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432252.198110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4621 00000100:00080000:2.0:1789432252.198113:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432252.198118:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432252.198122:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432252.198133:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913608448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432252.198140:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432252.198148:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432252.198152:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432252.198154:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432252.198159:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4621 00000100:00080000:2.0:1789432252.198162:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432252.198168:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432252.198173:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432252.198180:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432252/real 1789432252] req@ffff9811d2100000 x1876350913608192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432252.198188:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913608576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432252.198198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913608192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432252.198216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432252.198220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432252.198223:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913608192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432252.198241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432252.198260:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432252.198268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432252/real 1789432252] req@ffff9811d2103800 x1876350913608320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432252.198276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913608320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432252.198281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432252.198283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432252.198285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432252.198288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432252.198297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432252.198303:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432252/real 1789432252] req@ffff9811d2102d80 x1876350913608448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432252.198309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913608448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432252.198314:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432252.198315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432252.198317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432252.198320:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432252.198329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432252.198334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432252/real 1789432252] req@ffff9811d2101500 x1876350913608576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432252.198340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913608576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432252.198345:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432252.198347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432252.198348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432253.093962:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432254.117998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432255.141987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432256.166001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432257.189998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432257.317941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432257.317949:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432257.317952:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432257.317959:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4626 00000100:00080000:1.0:1789432257.317966:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432257.317974:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432257.318002:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913608704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432257.318023:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432257.318026:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432257.318027:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432257.318030:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4626 00000100:00080000:1.0:1789432257.318032:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432257.318035:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432257.318043:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432257.318051:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913608704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432257.318052:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913608832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432257.318064:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432257.318066:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432257.318067:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432257.318068:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4626 00000100:00080000:1.0:1789432257.318071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432257.318073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432257.318076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432257.318084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913608960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432257.318090:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432257.318092:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432257.318094:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432257.318096:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4626 00000100:00080000:1.0:1789432257.318098:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432257.318100:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432257.318104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432257.318107:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432257.318114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913609088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432257.318147:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432257/real 1789432257] req@ffff9811d0bded80 x1876350913608704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432257.318163:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913608704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432257.318184:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432257.318187:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432257.318190:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913608704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432257.318209:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432257.318224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432257.318232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432257/real 1789432257] req@ffff9811d0bdc000 x1876350913608832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432257.318239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913608832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432257.318244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432257.318246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432257.318248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432257.318251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432257.318260:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432257.318265:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432257/real 1789432257] req@ffff9811d0bdc700 x1876350913608960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432257.318276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913608960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432257.318280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432257.318282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432257.318283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913608960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432257.318286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432257.318302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432257.318308:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432257/real 1789432257] req@ffff9811d0bdd880 x1876350913609088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432257.318314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913609088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432257.318319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432257.318321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432257.318323:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432258.213970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432259.238051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432260.261985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432261.285966:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432262.309979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432262.437936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432262.437946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432262.437950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432262.437960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4632 00000100:00080000:3.0:1789432262.437968:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432262.437977:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432262.438011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913609216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432262.438035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432262.438039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432262.438041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432262.438045:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4632 00000100:00080000:3.0:1789432262.438049:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432262.438055:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432262.438065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432262.438070:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913609216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432262.438094:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913609344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432262.438106:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432262.438108:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432262.438109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432262.438111:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4632 00000100:00080000:3.0:1789432262.438116:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432262.438119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432262.438123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432262.438145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913609472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432262.438156:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432262.438161:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432262.438163:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432262.438167:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4632 00000100:00080000:3.0:1789432262.438171:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432262.438173:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609344:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789432262.438179:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432262.438185:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432262.438196:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913609600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432262.438211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432262.438237:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432263.333966:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432264.357993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432265.381999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432265.510203:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432265.510269:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432265.510289:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432265.510340:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432265.510350:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432265.510359:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:2.0:1789432265.510447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432262/real 1789432265] req@ffff9811d1bc5f80 x1876350913609600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432265.510479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913609600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432265.510497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432265.510503:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432265.510508:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609600:192.168.203.128@tcp:8:kworker.0 00000100:00100000:2.0:1789432265.510518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432262/real 1789432265] req@ffff9811d1bc4000 x1876350913609344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432265.510532:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913609344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432265.510540:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432265.510545:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432265.510548:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432265.510556:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432262/real 1789432265] req@ffff9811d1bc5880 x1876350913609472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432265.510567:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913609472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432265.510583:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432265.510586:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432265.510590:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432265.510598:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432262/real 1789432265] req@ffff9811d1bc7480 x1876350913609216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432265.510612:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913609216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432265.510620:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432265.510623:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432265.510626:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913609216:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432266.405957:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432267.430051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432267.557946:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432267.557962:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432267.557967:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432267.557979:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4637 00000100:00080000:2.0:1789432267.557987:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432267.557997:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432267.558029:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913609728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432267.558053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432267.558057:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432267.558059:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432267.558063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4637 00000100:00080000:2.0:1789432267.558067:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432267.558072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432267.558081:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432267.558109:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913609728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432267.558110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913609856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432267.558127:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432267.558130:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432267.558132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432267.558136:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4637 00000100:00080000:2.0:1789432267.558139:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432267.558144:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432267.558149:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432267.558172:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913609984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432267.558177:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432267.558187:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432267.558191:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432267.558193:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432267.558199:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4637 00000100:00080000:2.0:1789432267.558202:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432267.558207:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432267.558213:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432267.558219:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432267/real 1789432267] req@ffff9811d2103b80 x1876350913609728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432267.558238:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913610112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432267.558243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913609728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432267.558266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432267.558271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432267.558275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913609728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432267.558298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432267.558321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432267.558334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432267/real 1789432267] req@ffff9811d2101c00 x1876350913609856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432267.558346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913609856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432267.558359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432267.558363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432267.558366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432267.558371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432267.558390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432267.558402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432267/real 1789432267] req@ffff9811d2102a00 x1876350913609984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432267.558413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913609984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432267.558418:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432267.558421:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432267.558423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913609984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432267.558429:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432267.558445:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432267.558454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432267/real 1789432267] req@ffff9811d2100e00 x1876350913610112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432267.558462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913610112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432267.558475:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432267.558477:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432267.558480:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432268.453943:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432269.478007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432270.501929:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432271.525973:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432272.549987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432272.677939:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432272.677950:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432272.677954:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432272.677965:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4642 00000100:00080000:2.0:1789432272.677973:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432272.677984:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432272.678016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913610240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432272.678038:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432272.678041:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432272.678043:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432272.678047:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4642 00000100:00080000:2.0:1789432272.678050:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432272.678063:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432272.678079:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432272.678091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913610240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432272.678094:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913610368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432272.678102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432272.678105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432272.678107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432272.678110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4642 00000100:00080000:2.0:1789432272.678113:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432272.678116:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432272.678120:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432272.678128:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913610496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432272.678136:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432272.678139:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432272.678140:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432272.678142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432272.678143:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4642 00000100:00080000:2.0:1789432272.678146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432272.678149:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432272.678153:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432272.678161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913610624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432272.678173:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432272/real 1789432272] req@ffff9811d2101880 x1876350913610240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432272.678188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913610240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432272.678198:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432272.678202:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432272.678205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913610240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432272.678225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432272.678246:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432272.678256:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432272/real 1789432272] req@ffff9811d2101180 x1876350913610368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432272.678263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913610368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432272.678268:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432272.678269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432272.678271:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432272.678275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432272.678286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432272.678291:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432272/real 1789432272] req@ffff9811d2101f80 x1876350913610496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432272.678297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913610496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432272.678301:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432272.678302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432272.678304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432272.678307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610624:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432272.678316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432272.678321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432272/real 1789432272] req@ffff9811d2100e00 x1876350913610624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432272.678327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913610624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432272.678332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432272.678333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432272.678335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432273.573989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432274.597995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432275.622009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432276.645986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432277.669994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432277.797971:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432277.797981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432277.797986:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432277.797995:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4647 00000100:00080000:2.0:1789432277.798007:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432277.798017:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432277.798060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913610752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432277.798084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432277.798087:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432277.798089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432277.798093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4647 00000100:00080000:2.0:1789432277.798096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432277.798100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432277.798108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432277.798118:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913610880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432277.798128:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432277.798131:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913610752:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432277.798132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432277.798136:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432277.798142:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4647 00000100:00080000:2.0:1789432277.798145:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432277.798148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432277.798152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432277.798160:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913611008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432277.798169:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432277.798172:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432277.798174:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432277.798178:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4647 00000100:00080000:2.0:1789432277.798181:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789432277.798191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432277.798194:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432277.798200:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432277.798212:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913611136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432277.798229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432277/real 1789432277] req@ffff9811d2102a00 x1876350913610752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432277.798247:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913610752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432277.798257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432277.798261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432277.798264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913610752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432277.798293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432277.798313:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432277.798323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432277/real 1789432277] req@ffff9811d2101c00 x1876350913610880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432277.798333:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913610880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432277.798347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432277.798350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432277.798353:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913610880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432277.798359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432277.798378:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432277.798389:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432277/real 1789432277] req@ffff9811d2103b80 x1876350913611008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432277.798401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913611008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432277.798414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432277.798418:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432277.798422:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432277.798428:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432277.798447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432277.798458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432277/real 1789432277] req@ffff9811d2101500 x1876350913611136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432277.798469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913611136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432277.798477:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432277.798480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432277.798483:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432278.694047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432279.718078:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432280.742015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432281.766043:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432282.790030:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432282.917953:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432282.917962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432282.917966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432282.917975:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4652 00000100:00080000:1.0:1789432282.917993:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432282.918005:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432282.918038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913611264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432282.918065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789432282.918099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913611264:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432282.918111:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432282.918113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432282.918118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4652 00000100:00080000:1.0:1789432282.918122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432282.918127:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432282.918137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432282.918151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913611392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432282.918156:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432282.918166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432282.918171:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432282.918172:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432282.918177:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4652 00000100:00080000:1.0:1789432282.918188:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432282.918193:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432282.918197:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432282/real 1789432282] req@ffff9811d0bdf800 x1876350913611264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789432282.918200:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432282.918216:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913611520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432282.918222:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913611264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432282.918232:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432282.918237:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432282.918238:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432282.918240:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432282.918246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:1.0:1789432282.918247:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4652 00000100:00080000:1.0:1789432282.918255:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789432282.918257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913611264:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432282.918262:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432282.918276:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432282.918289:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913611648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432282.918293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432282.918324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432282.918334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432282/real 1789432282] req@ffff9811d0bdf100 x1876350913611392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432282.918342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913611392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432282.918347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432282.918349:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432282.918351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432282.918354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432282.918370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432282.918397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432282/real 1789432282] req@ffff9811d0bde300 x1876350913611520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432282.918409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913611520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432282.918417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432282.918420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432282.918422:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432282.918434:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432282.918463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432282.918476:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432282/real 1789432282] req@ffff9811d0bdf800 x1876350913611648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432282.918483:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913611648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432282.918488:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432282.918490:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432282.918492:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432283.813992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432284.837990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432285.861984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432286.885974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432287.909988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432288.037958:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432288.037968:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432288.037971:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432288.037980:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4657 00000100:00080000:2.0:1789432288.037988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432288.038000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432288.038034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913611776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432288.038055:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432288.038058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432288.038059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432288.038062:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4657 00000100:00080000:2.0:1789432288.038065:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432288.038068:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432288.038075:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432288.038083:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913611904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432288.038091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913611776:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432288.038092:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432288.038098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432288.038100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432288.038103:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4657 00000100:00080000:2.0:1789432288.038106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432288.038108:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432288.038111:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432288.038119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913612032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432288.038126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432288.038129:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432288.038131:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432288.038134:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4657 00000100:00080000:2.0:1789432288.038138:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789432288.038144:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432288.038151:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432288.038158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432288.038171:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913612160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432288.038200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432288/real 1789432288] req@ffff9811d2102d80 x1876350913611776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432288.038219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913611776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432288.038228:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432288.038231:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432288.038240:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913611776:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432288.038260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432288.038282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432288.038294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432288/real 1789432288] req@ffff9811d2103800 x1876350913611904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432288.038304:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913611904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432288.038312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432288.038315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432288.038318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913611904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432288.038323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432288.038340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432288.038350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432288/real 1789432288] req@ffff9811d2100000 x1876350913612032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432288.038360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913612032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432288.038366:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432288.038369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432288.038372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432288.038377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612160:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432288.038396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432288.038405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432288/real 1789432288] req@ffff9811d2101500 x1876350913612160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432288.038415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913612160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432288.038426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432288.038428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432288.038430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432288.933983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432289.957994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432290.982027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432292.006033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432293.030012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432293.157983:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432293.157993:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432293.157998:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432293.158008:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4662 00000100:00080000:2.0:1789432293.158018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432293.158031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432293.158070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913612288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432293.158099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432293.158102:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432293.158103:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432293.158106:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4662 00000100:00080000:2.0:1789432293.158108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432293.158111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432293.158122:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432293.158135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913612416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432293.158146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432293.158150:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913612288:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432293.158152:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432293.158159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432293.158167:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4662 00000100:00080000:2.0:1789432293.158170:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432293.158176:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432293.158181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432293.158197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913612544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432293.158209:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432293.158212:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432293.158215:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432293.158220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432293.158225:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4662 00000100:00080000:2.0:1789432293.158233:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432293.158242:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432293.158248:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432293.158256:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432293/real 1789432293] req@ffff9811d2103b80 x1876350913612288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432293.158265:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913612672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432293.158345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913612288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432293.158356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432293.158359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432293.158362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913612288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432293.158379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432293.158393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432293.158400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432293/real 1789432293] req@ffff9811d2101c00 x1876350913612416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432293.158407:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913612416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432293.158412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432293.158414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432293.158416:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432293.158419:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432293.158427:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432293.158432:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432293/real 1789432293] req@ffff9811d2102a00 x1876350913612544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432293.158438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913612544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432293.158446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432293.158448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432293.158450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432293.158453:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432293.158463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432293.158468:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432293/real 1789432293] req@ffff9811d2100e00 x1876350913612672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432293.158473:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913612672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432293.158478:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432293.158480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432293.158482:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432294.053998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432295.078060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432296.102227:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432297.126026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432298.150025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432298.277936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432298.277946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432298.277950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432298.277960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4667 00000100:00080000:2.0:1789432298.277969:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432298.277980:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432298.278018:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913612800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432298.278048:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432298.278051:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432298.278053:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432298.278057:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4667 00000100:00080000:2.0:1789432298.278060:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432298.278064:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432298.278074:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432298.278087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913612928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432298.278095:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913612800:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432298.278100:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432298.278104:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432298.278106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432298.278110:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4667 00000100:00080000:2.0:1789432298.278114:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432298.278118:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432298.278123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432298.278133:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913613056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432298.278143:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432298.278146:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432298.278148:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432298.278152:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4667 00000100:00080000:2.0:1789432298.278156:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789432298.278161:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432298.278171:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432298.278177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432298.278189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913613184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432298.278200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432298/real 1789432298] req@ffff9811d2101f80 x1876350913612800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432298.278220:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913612800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432298.278230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432298.278233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432298.278237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913612800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432298.278258:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432298.278274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432298.278283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432298/real 1789432298] req@ffff9811d2101180 x1876350913612928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432298.278290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913612928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432298.278295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432298.278297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432298.278299:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913612928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432298.278302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432298.278311:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432298.278316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432298/real 1789432298] req@ffff9811d2101880 x1876350913613056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432298.278325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913613056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432298.278332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432298.278335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432298.278338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432298.278344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613184:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432298.278361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432298.278371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432298/real 1789432298] req@ffff9811d2100e00 x1876350913613184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432298.278382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913613184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432298.278390:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432298.278393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432298.278395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432299.173982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432300.198001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432301.222040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432302.246089:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432303.270033:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432303.397963:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432303.397971:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432303.397974:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432303.397984:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4672 00000100:00080000:1.0:1789432303.397993:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432303.398001:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432303.398030:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913613312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432303.398050:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432303.398052:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432303.398054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432303.398056:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4672 00000100:00080000:1.0:1789432303.398058:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432303.398061:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432303.398068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432303.398076:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913613440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432303.398086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432303.398090:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432303.398091:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432303.398098:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4672 00000100:00100000:2.0:1789432303.398098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913613312:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432303.398103:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432303.398110:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432303.398115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432303.398127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913613568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432303.398136:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432303.398138:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432303.398140:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432303.398142:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4672 00000100:00080000:1.0:1789432303.398145:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432303.398148:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432303.398151:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432303.398157:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432303.398160:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913613696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432303.398200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432303/real 1789432303] req@ffff9811d0bdd880 x1876350913613312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432303.398217:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913613312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432303.398227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432303.398230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432303.398234:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913613312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432303.398241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432303.398261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432303.398274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432303/real 1789432303] req@ffff9811d0bdc700 x1876350913613440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432303.398286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913613440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432303.398294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432303.398298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432303.398301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432303.398322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432303.398340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432303.398350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432303/real 1789432303] req@ffff9811d0bdc000 x1876350913613568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432303.398364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913613568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432303.398371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432303.398374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432303.398377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432303.398383:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432303.398399:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432303.398409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432303/real 1789432303] req@ffff9811d0bded80 x1876350913613696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432303.398420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913613696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432303.398427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432303.398429:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432303.398432:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432304.293979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432305.318003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432306.341960:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432307.365955:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432308.389976:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432308.517907:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432308.517917:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432308.517921:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432308.517930:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4678 00000100:00080000:1.0:1789432308.517938:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432308.517948:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432308.517977:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913613824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432308.518013:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432308.518016:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432308.518018:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432308.518022:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4678 00000100:00080000:1.0:1789432308.518025:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432308.518029:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432308.518039:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432308.518051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913613952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432308.518055:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913613824:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432308.518060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432308.518064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432308.518065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432308.518067:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4678 00000100:00080000:1.0:1789432308.518069:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432308.518072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432308.518075:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432308.518085:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913614080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432308.518094:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432308.518098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432308.518101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432308.518104:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4678 00000100:00080000:1.0:1789432308.518108:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432308.518112:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432308.518117:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:2.0:1789432308.518118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432308.518131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913614208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432308.518155:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432308/real 1789432308] req@ffff9811d0bdf480 x1876350913613824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432308.518172:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913613824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432308.518182:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432308.518185:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432308.518189:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913613824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432308.518207:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432308.518226:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432308.518238:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432308/real 1789432308] req@ffff9811d0bdfb80 x1876350913613952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432308.518249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913613952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432308.518257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432308.518260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432308.518263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913613952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432308.518268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432308.518284:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432308.518293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432308/real 1789432308] req@ffff9811d0bdea00 x1876350913614080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432308.518303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913614080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432308.518310:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432308.518312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432308.518316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432308.518321:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432308.518337:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432308.518346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432308/real 1789432308] req@ffff9811d0bde680 x1876350913614208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432308.518356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913614208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432308.518363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432308.518366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432308.518369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432309.413965:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432310.437990:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432311.461991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432312.485969:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432313.509972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432313.637934:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432313.637944:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432313.637948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432313.637957:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4683 00000100:00080000:3.0:1789432313.637964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432313.637972:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432313.637999:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913614336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432313.638024:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432313.638026:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432313.638027:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432313.638030:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4683 00000100:00080000:3.0:1789432313.638032:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432313.638035:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432313.638042:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432313.638051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913614464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432313.638058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432313.638060:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432313.638062:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789432313.638066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913614336:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432313.638067:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4683 00000100:00080000:3.0:1789432313.638070:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432313.638075:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432313.638078:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432313.638085:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913614592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432313.638092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432313.638094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432313.638095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432313.638098:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4683 00000100:00080000:3.0:1789432313.638099:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432313.638102:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432313.638104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432313.638110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913614720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432313.638131:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432313.638173:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432313/real 1789432313] req@ffff9811d1bc6300 x1876350913614336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432313.638190:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913614336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432313.638198:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432313.638200:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432313.638203:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913614336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432313.638219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432313.638236:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432313.638244:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432313/real 1789432313] req@ffff9811d1bc5500 x1876350913614464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432313.638251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913614464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432313.638256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432313.638258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432313.638260:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432313.638263:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432313.638271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432313.638276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432313/real 1789432313] req@ffff9811d1bc4700 x1876350913614592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432313.638283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913614592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432313.638290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432313.638293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432313.638296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432313.638301:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614720:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432313.638321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432313.638329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432313/real 1789432313] req@ffff9811d1bc7100 x1876350913614720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432313.638340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913614720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432313.638345:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432313.638347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432313.638348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432314.533991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432315.557979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432316.582050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432317.606065:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432318.629984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432318.757929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432318.757936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432318.757939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432318.757946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4688 00000100:00080000:1.0:1789432318.757953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432318.757962:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432318.757989:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913614848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432318.758009:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432318.758011:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432318.758013:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432318.758015:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4688 00000100:00080000:1.0:1789432318.758017:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432318.758019:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432318.758026:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432318.758033:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913614976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432318.758040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432318.758043:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432318.758044:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432318.758046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4688 00000100:00080000:1.0:1789432318.758048:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432318.758050:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432318.758053:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432318.758061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913615104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789432318.758061:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913614848:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432318.758073:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432318.758075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432318.758076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432318.758078:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4688 00000100:00080000:1.0:1789432318.758080:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432318.758083:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432318.758085:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432318.758092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913615232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789432318.758114:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432318.758143:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432318/real 1789432318] req@ffff9811d0bdd180 x1876350913614848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432318.758155:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913614848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432318.758162:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432318.758164:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432318.758167:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913614848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432318.758171:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432318.758184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432318.758190:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432318/real 1789432318] req@ffff9811d0bdd500 x1876350913614976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432318.758197:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913614976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432318.758205:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432318.758207:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432318.758209:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913614976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432318.758223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789432318.758231:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432318.758236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432318/real 1789432318] req@ffff9811d0bddc00 x1876350913615104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432318.758242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913615104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432318.758246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432318.758248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432318.758249:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789432318.758252:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789432318.758261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432318.758266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432318/real 1789432318] req@ffff9811d0bdc380 x1876350913615232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432318.758276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913615232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432318.758283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789432318.758286:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789432318.758289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432319.653989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432320.677971:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432321.702019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432322.725969:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432323.750032:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432323.877957:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432323.877966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432323.877970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432323.877981:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4693 00000100:00080000:2.0:1789432323.877990:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432323.878006:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432323.878047:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913615360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432323.878071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432323.878074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432323.878075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432323.878078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4693 00000100:00080000:2.0:1789432323.878081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432323.878086:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432323.878097:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432323.878111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913615360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432323.878113:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913615488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432323.878127:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432323.878131:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432323.878133:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432323.878137:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4693 00000100:00080000:2.0:1789432323.878140:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432323.878145:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432323.878149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432323.878159:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913615616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432323.878166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432323.878169:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432323.878170:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432323.878172:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4693 00000100:00080000:2.0:1789432323.878182:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432323.878185:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432323.878188:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432323.878197:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432323.878197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913615744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432323.878246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432323/real 1789432323] req@ffff9811d2102a00 x1876350913615360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432323.878267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913615360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432323.878275:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432323.878278:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432323.878281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913615360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432323.878301:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432323.878318:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432323.878326:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432323/real 1789432323] req@ffff9811d2101c00 x1876350913615488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432323.878334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913615488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432323.878342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432323.878352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432323.878355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432323.878361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432323.878381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432323.878390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432323/real 1789432323] req@ffff9811d2103b80 x1876350913615616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432323.878398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913615616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432323.878407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432323.878409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432323.878411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432323.878414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432323.878427:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432323.878432:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432323/real 1789432323] req@ffff9811d2101500 x1876350913615744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432323.878439:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913615744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432323.878444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432323.878446:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432323.878447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913615744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432324.773954:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432325.797999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432326.821955:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432327.845980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432328.869936:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432328.997929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432328.997936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432328.997939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432328.997948:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4698 00000100:00080000:3.0:1789432328.997955:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432328.997964:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432328.997994:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913615872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432328.998014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432328.998016:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432328.998017:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432328.998020:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4698 00000100:00080000:3.0:1789432328.998022:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432328.998025:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432328.998031:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432328.998045:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913616000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432328.998052:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432328.998054:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432328.998055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432328.998060:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913615872:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432328.998061:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4698 00000100:00080000:3.0:1789432328.998066:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432328.998070:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432328.998073:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432328.998080:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913616128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432328.998087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432328.998089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432328.998090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432328.998092:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4698 00000100:00080000:3.0:1789432328.998095:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432328.998097:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432328.998100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432328.998106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913616256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432328.998141:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432328.998159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432328.998170:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432329.893984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432330.917978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432331.941978:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432332.070101:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432332.070137:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432332.070151:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432332.070190:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432332.070199:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432332.070206:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432332.070249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432328/real 1789432332] req@ffff9811d1bc7100 x1876350913616000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432332.070286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913616000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432332.070301:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432332.070305:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432332.070309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432332.070317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432328/real 1789432332] req@ffff9811d1bc4700 x1876350913616128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432332.070328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913616128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432332.070335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432332.070338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432332.070341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432332.070348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432328/real 1789432332] req@ffff9811d1bc5500 x1876350913616256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432332.070359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913616256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432332.070367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432332.070370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432332.070374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616256:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432332.070380:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432328/real 1789432332] req@ffff9811d1bc6680 x1876350913615872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432332.070392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913615872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432332.070400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432332.070403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432332.070406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913615872:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432332.966057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432333.990058:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432334.117952:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432334.117965:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432334.117970:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432334.117982:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4703 00000100:00080000:2.0:1789432334.117993:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432334.118008:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432334.118048:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913616384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432334.118076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432334.118080:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432334.118082:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432334.118086:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4703 00000100:00080000:2.0:1789432334.118090:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432334.118096:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432334.118108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432334.118122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913616384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432334.118125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913616512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432334.118139:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432334.118142:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432334.118143:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432334.118148:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4703 00000100:00080000:2.0:1789432334.118151:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432334.118157:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432334.118162:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432334.118182:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913616640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432334.118194:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432334.118197:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432334.118199:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432334.118203:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4703 00000100:00080000:2.0:1789432334.118207:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432334.118212:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789432334.118217:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432334.118220:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432334.118236:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913616768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432334.118270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432334/real 1789432334] req@ffff9811d2100000 x1876350913616384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432334.118292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913616384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432334.118303:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432334.118307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432334.118311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913616384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432334.118340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432334.118369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432334.118382:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432334/real 1789432334] req@ffff9811d2103800 x1876350913616512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432334.118399:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913616512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432334.118407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432334.118410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432334.118413:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432334.118418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432334.118435:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432334.118445:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432334/real 1789432334] req@ffff9811d2102d80 x1876350913616640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432334.118456:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913616640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432334.118464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432334.118468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432334.118471:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432334.118476:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432334.118494:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432334.118503:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432334/real 1789432334] req@ffff9811d2101500 x1876350913616768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432334.118514:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913616768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432334.118522:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432334.118525:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432334.118528:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913616768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432335.014047:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432336.037985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432337.062076:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432338.086032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432339.110054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432339.237958:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432339.237973:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432339.237979:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432339.237993:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4708 00000100:00080000:2.0:1789432339.238006:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432339.238021:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432339.238074:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913616896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432339.238107:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432339.238112:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432339.238114:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432339.238120:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4708 00000100:00080000:2.0:1789432339.238123:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432339.238128:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432339.238139:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432339.238154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913617024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432339.238155:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913616896:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432339.238170:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432339.238173:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432339.238175:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432339.238178:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4708 00000100:00080000:2.0:1789432339.238180:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432339.238182:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432339.238186:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432339.238194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913617152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432339.238202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432339.238205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432339.238206:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432339.238209:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4708 00000100:00080000:2.0:1789432339.238226:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432339.238229:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432339.238232:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432339.238237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432339.238243:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913617280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432339.238294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432339/real 1789432339] req@ffff9811d2103b80 x1876350913616896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432339.238318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913616896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432339.238331:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432339.238336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432339.238340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913616896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432339.238368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432339.238394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432339.238409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432339/real 1789432339] req@ffff9811d2101c00 x1876350913617024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432339.238418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913617024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432339.238429:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432339.238432:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432339.238434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432339.238439:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432339.238453:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432339.238459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432339/real 1789432339] req@ffff9811d2102a00 x1876350913617152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432339.238467:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913617152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432339.238472:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432339.238474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432339.238476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432339.238480:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617280:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432339.238491:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432339.238497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432339/real 1789432339] req@ffff9811d2100e00 x1876350913617280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432339.238505:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913617280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432339.238510:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432339.238511:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432339.238514:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432340.134103:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432341.158050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432342.182083:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432343.206104:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432344.230044:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432344.357970:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432344.357983:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432344.357988:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432344.358000:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4713 00000100:00080000:1.0:1789432344.358009:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432344.358021:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432344.358070:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913617408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432344.358095:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432344.358098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432344.358100:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432344.358103:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4713 00000100:00080000:1.0:1789432344.358106:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432344.358110:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432344.358120:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432344.358130:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913617536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432344.358139:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432344.358144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432344.358149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789432344.358151:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913617408:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432344.358154:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4713 00000100:00080000:1.0:1789432344.358159:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432344.358167:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432344.358171:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432344.358183:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913617664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432344.358191:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432344.358195:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432344.358196:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432344.358199:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4713 00000100:00080000:1.0:1789432344.358201:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432344.358211:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432344.358215:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432344.358223:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913617792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432344.358233:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432344.358276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432344/real 1789432344] req@ffff9811d0bdc380 x1876350913617408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432344.358291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913617408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432344.358299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432344.358302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432344.358305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913617408:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432344.358324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432344.358340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432344.358348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432344/real 1789432344] req@ffff9811d0bddc00 x1876350913617536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432344.358356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913617536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432344.358362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432344.358364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432344.358366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432344.358372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432344.358392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432344.358404:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432344/real 1789432344] req@ffff9811d0bdd500 x1876350913617664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432344.358417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913617664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432344.358425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432344.358428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432344.358430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432344.358435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617792:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432344.358451:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432344.358458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432344/real 1789432344] req@ffff9811d0bdd180 x1876350913617792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432344.358466:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913617792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432344.358471:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432344.358474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432344.358476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913617792:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432345.254066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432346.277990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432347.302007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432348.325944:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432349.349993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432349.477942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432349.477953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432349.477957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432349.477968:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4719 00000100:00080000:3.0:1789432349.477979:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432349.477996:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432349.478038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913617920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432349.478064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432349.478067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432349.478068:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432349.478072:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4719 00000100:00080000:3.0:1789432349.478074:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432349.478078:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432349.478087:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432349.478098:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913618048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432349.478105:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913617920:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432349.478117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432349.478121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432349.478123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432349.478129:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4719 00000100:00080000:3.0:1789432349.478133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432349.478139:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432349.478145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432349.478159:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913618176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432349.478172:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432349.478175:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432349.478177:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432349.478183:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4719 00000800:00000100:1.0:1789432349.478185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432349.478188:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432349.478193:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432349.478198:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432349.478208:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913618304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432349.478232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432349/real 1789432349] req@ffff9811d1bc6300 x1876350913617920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432349.478248:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913617920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432349.478257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432349.478261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432349.478264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913617920:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432349.478285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432349.478301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432349.478310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432349/real 1789432349] req@ffff9811d1bc7480 x1876350913618048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432349.478318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913618048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432349.478323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432349.478326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432349.478328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432349.478332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432349.478342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432349.478348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432349/real 1789432349] req@ffff9811d1bc5880 x1876350913618176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432349.478355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913618176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432349.478360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432349.478362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432349.478364:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432349.478368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618304:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432349.478380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432349.478386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432349/real 1789432349] req@ffff9811d1bc4000 x1876350913618304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432349.478393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913618304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432349.478398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432349.478400:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432349.478402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618304:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432350.373975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432351.397985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432352.422013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432353.446048:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432354.470001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432354.597968:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432354.597978:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432354.597983:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432354.597993:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4724 00000100:00080000:3.0:1789432354.598003:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432354.598016:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432354.598056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913618432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432354.598083:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432354.598086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432354.598087:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432354.598090:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4724 00000100:00080000:3.0:1789432354.598093:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432354.598096:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432354.598106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432354.598120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913618560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432354.598126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432354.598129:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432354.598130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432354.598131:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4724 00000100:00080000:3.0:1789432354.598133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432354.598135:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432354.598138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432354.598144:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913618688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432354.598150:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432354.598153:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432354.598155:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432354.598157:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4724 00000100:00080000:3.0:1789432354.598159:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432354.598161:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432354.598164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432354.598171:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913618432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432354.598172:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913618816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432354.598244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432354.598291:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432354/real 1789432354] req@ffff9811d1bc5f80 x1876350913618432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432354.598308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913618432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432354.598317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432354.598320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432354.598322:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913618432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432354.598327:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432354.598341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432354.598348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432354/real 1789432354] req@ffff9811d1bc5c00 x1876350913618560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432354.598360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913618560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432354.598364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432354.598366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432354.598367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432354.598371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432354.598380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432354.598385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432354/real 1789432354] req@ffff9811d1bc7800 x1876350913618688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432354.598391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913618688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432354.598396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432354.598397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432354.598399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432354.598420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618816:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432354.598430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432354.598435:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432354/real 1789432354] req@ffff9811d1bc5180 x1876350913618816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432354.598441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913618816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432354.598450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432354.598452:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432354.598454:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913618816:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432355.494091:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432356.518079:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432357.542066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432358.566086:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432359.590083:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432359.717988:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432359.718000:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432359.718005:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432359.718017:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4729 00000100:00080000:1.0:1789432359.718034:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432359.718047:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432359.718098:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913618944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432359.718130:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432359.718135:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432359.718137:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432359.718143:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4729 00000100:00080000:1.0:1789432359.718147:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432359.718152:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432359.718163:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913618944:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789432359.718164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432359.718181:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913619072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432359.718192:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432359.718196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432359.718198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432359.718202:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4729 00000100:00080000:1.0:1789432359.718206:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432359.718211:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432359.718217:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432359.718228:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913619200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432359.718237:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432359.718243:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1789432359.718243:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432359.718247:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432359.718253:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4729 00000100:00080000:1.0:1789432359.718258:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432359.718263:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432359.718268:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432359.718280:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913619328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432359.718288:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432359/real 1789432359] req@ffff9811d0bde680 x1876350913618944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432359.718306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913618944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432359.718318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432359.718322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432359.718326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913618944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432359.718352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432359.718370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432359.718379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432359/real 1789432359] req@ffff9811d0bdea00 x1876350913619072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432359.718387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913619072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432359.718392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432359.718394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432359.718397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432359.718400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432359.718415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432359.718572:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432359/real 1789432359] req@ffff9811d0bdfb80 x1876350913619200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432359.718587:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913619200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432359.718594:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432359.718597:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432359.718600:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432359.718606:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619328:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432359.718627:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432359.718638:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432359/real 1789432359] req@ffff9811d0bdf480 x1876350913619328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432359.718647:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913619328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432359.718655:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432359.718657:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432359.718660:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619328:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432360.614053:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432361.638019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432362.661997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432363.686005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432364.710013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432364.837939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432364.837950:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432364.837955:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432364.837967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4734 00000100:00080000:3.0:1789432364.837974:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432364.837984:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432364.838019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913619456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432364.838041:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432364.838044:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432364.838046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432364.838049:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4734 00000100:00080000:3.0:1789432364.838052:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432364.838056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432364.838064:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432364.838075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913619456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432364.838082:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913619584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432364.838096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432364.838100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432364.838102:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432364.838107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4734 00000100:00080000:3.0:1789432364.838112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432364.838117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432364.838123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789432364.838130:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432364.838140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913619712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432364.838154:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432364.838156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432364.838158:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432364.838161:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4734 00000100:00100000:1.0:1789432364.838166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432364/real 1789432364] req@ffff9811d1bc6a00 x1876350913619456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432364.838166:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432364.838174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432364.838181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432364.838207:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913619456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432364.838210:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913619840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432364.838241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432364.838247:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432364.838253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913619456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432364.838277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432364.838304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432364.838320:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432364/real 1789432364] req@ffff9811d1bc5180 x1876350913619584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432364.838335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913619584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432364.838342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432364.838345:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432364.838348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432364.838356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432364.838379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432364.838411:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432364/real 1789432364] req@ffff9811d1bc7800 x1876350913619712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432364.838428:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913619712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432364.838452:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432364.838455:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432364.838459:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432364.838465:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619840:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432364.838490:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432364.838502:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432364/real 1789432364] req@ffff9811d1bc5c00 x1876350913619840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432364.838511:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913619840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432364.838517:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432364.838519:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432364.838521:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913619840:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432365.733974:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432366.758008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432367.781950:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432368.805970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432369.830035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432369.957964:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432369.957974:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432369.957979:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432369.957991:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4739 00000100:00080000:2.0:1789432369.958002:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432369.958017:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432369.958067:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913619968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432369.958098:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432369.958101:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432369.958103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432369.958107:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4739 00000100:00080000:2.0:1789432369.958110:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432369.958114:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432369.958124:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432369.958135:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913620096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432369.958144:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432369.958147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432369.958149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432369.958152:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4739 00000100:00080000:2.0:1789432369.958155:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432369.958157:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432369.958160:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432369.958170:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913620224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432369.958179:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432369.958183:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432369.958185:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432369.958187:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913619968:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432369.958195:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4739 00000100:00080000:2.0:1789432369.958206:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432369.958210:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432369.958213:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432369.958230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913620352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432369.958275:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432369.958317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432369/real 1789432369] req@ffff9811d2101880 x1876350913619968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432369.958336:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913619968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432369.958348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432369.958368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432369.958372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913619968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432369.958381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432369.958404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432369.958416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432369/real 1789432369] req@ffff9811d2101180 x1876350913620096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432369.958437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913620096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432369.958446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432369.958450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432369.958454:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432369.958478:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432369.958494:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432369.958504:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432369/real 1789432369] req@ffff9811d2101f80 x1876350913620224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432369.958516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913620224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432369.958525:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432369.958528:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432369.958532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432369.958538:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620352:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432369.958553:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432369.958562:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432369/real 1789432369] req@ffff9811d2100e00 x1876350913620352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432369.958573:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913620352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432369.958589:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432369.958592:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432369.958596:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620352:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432370.854070:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432371.878019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432372.901997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432373.926009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432374.950026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432375.077926:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432375.077935:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432375.077938:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432375.077946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4744 00000100:00080000:1.0:1789432375.077953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432375.077963:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432375.077994:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913620480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432375.078018:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432375.078021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432375.078023:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432375.078026:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4744 00000100:00080000:1.0:1789432375.078028:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432375.078031:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432375.078038:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432375.078048:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913620608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432375.078059:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913620480:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432375.078067:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432375.078071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432375.078073:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432375.078076:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4744 00000100:00080000:1.0:1789432375.078078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432375.078081:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432375.078085:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432375.078095:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913620736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432375.078103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432375.078105:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432375.078106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432375.078108:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4744 00000100:00080000:1.0:1789432375.078111:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789432375.078116:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432375.078124:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432375.078131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432375.078145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913620864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432375.078151:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432375/real 1789432375] req@ffff9811d0bded80 x1876350913620480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432375.078173:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913620480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432375.078183:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432375.078187:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432375.078192:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913620480:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432375.078215:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432375.078240:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432375.078251:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432375/real 1789432375] req@ffff9811d0bdc000 x1876350913620608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432375.078260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913620608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432375.078266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432375.078268:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432375.078270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432375.078275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432375.078287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432375.078293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432375/real 1789432375] req@ffff9811d0bdc700 x1876350913620736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432375.078306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913620736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432375.078324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432375.078328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432375.078332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432375.078338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620864:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432375.078359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432375.078371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432375/real 1789432375] req@ffff9811d0bdd880 x1876350913620864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432375.078387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913620864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432375.078395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432375.078399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432375.078402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913620864:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432375.973990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432376.997980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432378.022097:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432379.046013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432380.070017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432380.197984:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432380.197995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432380.197999:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432380.198009:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4749 00000100:00080000:0.0:1789432380.198018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432380.198031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432380.198067:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913620992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432380.198095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432380.198099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432380.198101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432380.198107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4749 00000100:00080000:0.0:1789432380.198112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432380.198118:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432380.198124:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913620992:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789432380.198130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432380.198147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913621120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432380.198159:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432380.198163:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432380.198165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432380.198170:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4749 00000100:00080000:0.0:1789432380.198174:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432380.198179:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432380.198184:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789432380.198188:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432380.198199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913621248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432380.198212:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432380.198216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432380.198218:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432380.198224:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4749 00000100:00100000:3.0:1789432380.198227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432380/real 1789432380] req@ffff9811c8cd5f80 x1876350913620992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432380.198229:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432380.198239:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432380.198245:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432380.198249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913620992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432380.198261:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913621376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432380.198263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432380.198273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432380.198295:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913620992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432380.198320:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432380.198354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432380.198366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432380/real 1789432380] req@ffff9811c8cd6d80 x1876350913621120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432380.198375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913621120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432380.198382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432380.198384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432380.198387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432380.198391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432380.198404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432380.198411:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432380/real 1789432380] req@ffff9811c8cd5500 x1876350913621248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432380.198418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913621248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432380.198423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432380.198425:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432380.198427:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432380.198431:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621376:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432380.198442:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432380.198448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432380/real 1789432380] req@ffff9811c8cd4a80 x1876350913621376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432380.198456:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913621376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432380.198461:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432380.198463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432380.198465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621376:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432381.094027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432382.118030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432383.142006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432384.166070:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432385.189970:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432385.317953:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432385.317962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432385.317966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432385.317976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4754 00000100:00080000:0.0:1789432385.317986:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432385.317997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432385.318035:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913621504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432385.318062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432385.318065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432385.318066:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432385.318069:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4754 00000100:00080000:0.0:1789432385.318072:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432385.318075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432385.318084:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432385.318092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913621632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432385.318099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432385.318103:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432385.318105:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432385.318109:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4754 00000100:00100000:3.0:1789432385.318110:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913621504:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432385.318111:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432385.318115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432385.318123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432385.318132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913621760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432385.318139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432385.318141:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432385.318143:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432385.318146:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4754 00000100:00080000:0.0:1789432385.318149:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432385.318152:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432385.318157:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432385.318167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913621888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432385.318184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432385.318226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432385/real 1789432385] req@ffff9811c8cd6a00 x1876350913621504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432385.318249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913621504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432385.318257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432385.318260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432385.318263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913621504:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432385.318280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432385.318302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432385.318310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432385/real 1789432385] req@ffff9811c8cd7b80 x1876350913621632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432385.318317:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913621632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432385.318322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432385.318325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432385.318326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432385.318330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432385.318340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432385.318345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432385/real 1789432385] req@ffff9811c8cd6300 x1876350913621760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432385.318351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913621760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432385.318356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432385.318357:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432385.318359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432385.318362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621888:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432385.318377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432385.318383:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432385/real 1789432385] req@ffff9811c8cd7480 x1876350913621888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432385.318389:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913621888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432385.318394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432385.318395:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432385.318397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913621888:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432386.214017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432387.238275:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432388.262002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432389.285996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432390.310015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432390.437923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432390.437932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432390.437937:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432390.437946:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4760 00000100:00080000:1.0:1789432390.437955:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432390.437966:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432390.438002:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913622016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432390.438031:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432390.438034:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432390.438036:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432390.438041:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4760 00000100:00080000:1.0:1789432390.438044:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432390.438049:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432390.438057:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913622016:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789432390.438058:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432390.438073:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913622144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432390.438082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432390.438086:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432390.438087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432390.438091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4760 00000100:00080000:1.0:1789432390.438093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432390.438097:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432390.438101:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432390.438111:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913622272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432390.438120:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789432390.438124:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432390.438125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432390.438128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432390.438133:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4760 00000100:00080000:1.0:1789432390.438137:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432390.438142:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432390.438147:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432390.438156:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432390/real 1789432390] req@ffff9811d0bdf800 x1876350913622016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432390.438161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913622400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432390.438214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913622016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432390.438225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432390.438229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432390.438234:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913622016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432390.438258:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432390.438280:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432390.438292:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432390/real 1789432390] req@ffff9811d0bde300 x1876350913622144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432390.438303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913622144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432390.438311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432390.438315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432390.438318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432390.438323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432390.438342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432390.438353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432390/real 1789432390] req@ffff9811d0bdf100 x1876350913622272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432390.438364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913622272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432390.438376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432390.438379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432390.438382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432390.438388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622400:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432390.438407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432390.438435:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432390/real 1789432390] req@ffff9811d0bdd880 x1876350913622400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432390.438447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913622400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432390.438455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432390.438457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432390.438460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622400:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432391.333979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432392.357985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432393.381985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432394.406185:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432395.429984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432395.557937:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432395.557945:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432395.557949:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432395.557959:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4765 00000100:00080000:3.0:1789432395.557966:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432395.557978:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432395.558014:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913622528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432395.558039:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432395.558044:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432395.558045:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432395.558071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913622528:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432395.558107:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4765 00000100:00080000:3.0:1789432395.558113:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432395.558120:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432395.558130:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432395.558145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913622656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432395.558155:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432395.558158:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432395.558160:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432395.558173:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4765 00000100:00100000:1.0:1789432395.558176:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622656:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789432395.558178:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432395.558187:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432395.558193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432395.558207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913622784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432395.558229:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432395.558233:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432395.558235:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432395.558240:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4765 00000100:00080000:3.0:1789432395.558244:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789432395.558246:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622784:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789432395.558252:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432395.558259:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432395.558274:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913622912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432395.558302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622912:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432396.453975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432397.478027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432398.502034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432398.630230:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432398.630286:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432398.630312:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432398.630359:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432398.630369:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432398.630377:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432398.630472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432395/real 1789432398] req@ffff9811d1bc7480 x1876350913622912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432398.630506:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913622912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432398.630518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432398.630522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432398.630525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622912:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432398.630533:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432395/real 1789432398] req@ffff9811d1bc5880 x1876350913622784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432398.630546:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913622784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432398.630560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432398.630564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432398.630567:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432398.630574:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432395/real 1789432398] req@ffff9811d1bc4000 x1876350913622656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432398.630585:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913622656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432398.630593:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432398.630596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432398.630599:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913622656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432398.630611:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432395/real 1789432398] req@ffff9811d1bc5f80 x1876350913622528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432398.630623:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913622528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432398.630629:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432398.630632:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432398.630635:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913622528:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432399.525982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432400.549983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432400.677940:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432400.677950:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432400.677954:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432400.677964:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4770 00000100:00080000:1.0:1789432400.677980:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432400.677992:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432400.678022:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913623040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432400.678051:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432400.678054:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432400.678057:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432400.678061:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4770 00000100:00080000:1.0:1789432400.678064:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432400.678069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432400.678078:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432400.678090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913623168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432400.678099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432400.678101:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432400.678103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432400.678106:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4770 00000100:00080000:1.0:1789432400.678109:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432400.678113:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432400.678117:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432400.678126:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913623296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432400.678145:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432400.678149:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432400.678151:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432400.678156:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4770 00000100:00080000:1.0:1789432400.678160:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789432400.678172:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913623040:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432400.678178:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432400.678185:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432400.678201:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913623424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432400.678239:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432400.678277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432400/real 1789432400] req@ffff9811d0bdc700 x1876350913623040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432400.678295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913623040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432400.678305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432400.678309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432400.678313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913623040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432400.678320:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432400.678346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432400.678358:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432400/real 1789432400] req@ffff9811d0bdc000 x1876350913623168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432400.678370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913623168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432400.678378:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432400.678382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432400.678385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432400.678396:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432400.678414:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432400.678424:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432400/real 1789432400] req@ffff9811d0bded80 x1876350913623296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432400.678446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913623296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432400.678462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432400.678466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432400.678469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432400.678492:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623424:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432400.678511:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432400.678521:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432400/real 1789432400] req@ffff9811d0bdf480 x1876350913623424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432400.678532:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913623424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432400.678541:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432400.678544:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432400.678547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623424:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432401.573989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432402.597986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432403.622008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432404.645991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432405.670011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432405.797980:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432405.797987:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432405.797990:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432405.797997:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4775 00000100:00080000:3.0:1789432405.798002:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432405.798012:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432405.798038:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913623552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432405.798056:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432405.798058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432405.798059:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432405.798062:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4775 00000100:00080000:3.0:1789432405.798064:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432405.798066:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432405.798073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432405.798080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913623680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432405.798103:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913623552:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432405.798107:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432405.798112:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432405.798114:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432405.798116:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4775 00000100:00080000:3.0:1789432405.798119:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432405.798121:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432405.798125:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432405.798133:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913623808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432405.798140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432405.798142:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432405.798144:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432405.798146:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4775 00000100:00080000:3.0:1789432405.798148:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432405.798153:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432405.798156:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432405.798161:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432405.798164:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913623936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432405.798191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432405/real 1789432405] req@ffff9811d1bc6300 x1876350913623552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432405.798204:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913623552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432405.798212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432405.798214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432405.798217:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913623552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432405.798231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432405.798244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432405.798251:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432405/real 1789432405] req@ffff9811d1bc6680 x1876350913623680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432405.798258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913623680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432405.798263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432405.798265:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432405.798267:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432405.798270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432405.798278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432405.798283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432405/real 1789432405] req@ffff9811d1bc5500 x1876350913623808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432405.798289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913623808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432405.798294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432405.798295:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432405.798297:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432405.798300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623936:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432405.798309:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432405.798314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432405/real 1789432405] req@ffff9811d1bc4700 x1876350913623936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432405.798320:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913623936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432405.798324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432405.798326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432405.798328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913623936:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432406.693952:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432407.717998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432408.741919:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432409.766001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432410.789997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432410.917981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432410.917989:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432410.917992:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432410.917999:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4780 00000100:00080000:2.0:1789432410.918007:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432410.918016:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432410.918044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913624064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432410.918075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432410.918079:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432410.918085:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432410.918090:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4780 00000100:00080000:2.0:1789432410.918093:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432410.918098:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432410.918107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432410.918120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913624192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432410.918127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913624064:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432410.918135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432410.918140:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432410.918142:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432410.918146:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4780 00000100:00080000:2.0:1789432410.918149:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432410.918153:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432410.918157:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432410.918166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913624320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432410.918173:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432410.918176:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432410.918178:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432410.918180:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4780 00000100:00080000:2.0:1789432410.918182:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432410.918185:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432410.918190:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432410.918191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432410.918200:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913624448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432410.918233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432410/real 1789432410] req@ffff9811d2102a00 x1876350913624064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432410.918250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913624064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432410.918260:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432410.918264:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432410.918268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913624064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432410.918288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432410.918310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432410.918322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432410/real 1789432410] req@ffff9811d2101c00 x1876350913624192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432410.918334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913624192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432410.918343:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432410.918346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432410.918349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432410.918355:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432410.918374:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432410.918385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432410/real 1789432410] req@ffff9811d2103b80 x1876350913624320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432410.918406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913624320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432410.918414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432410.918416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432410.918420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432410.918425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624448:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432410.918447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432410.918458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432410/real 1789432410] req@ffff9811d2101500 x1876350913624448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432410.918469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913624448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432410.918477:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432410.918480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432410.918483:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624448:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432411.813973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432412.837997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432413.862016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432414.886001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432415.909988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432416.037947:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432416.037955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432416.037958:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432416.037965:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4785 00000100:00080000:1.0:1789432416.037972:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432416.037981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432416.038014:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913624576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432416.038049:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432416.038053:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432416.038055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432416.038059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4785 00000100:00080000:1.0:1789432416.038062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432416.038067:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432416.038076:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432416.038087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913624704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432416.038092:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913624576:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432416.038096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432416.038100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432416.038101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432416.038104:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4785 00000100:00080000:1.0:1789432416.038106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432416.038109:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432416.038112:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432416.038119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913624832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432416.038126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432416.038128:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432416.038129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432416.038132:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4785 00000100:00080000:1.0:1789432416.038135:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432416.038137:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432416.038140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432416.038146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913624960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432416.038155:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432416.038192:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432416/real 1789432416] req@ffff9811d0bdfb80 x1876350913624576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432416.038210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913624576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432416.038221:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432416.038225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432416.038229:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913624576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432416.038250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432416.038271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432416.038283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432416/real 1789432416] req@ffff9811d0bdea00 x1876350913624704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432416.038294:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913624704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432416.038308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432416.038311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432416.038315:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432416.038321:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432416.038339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432416.038348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432416/real 1789432416] req@ffff9811d0bde680 x1876350913624832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432416.038359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913624832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432416.038367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432416.038369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432416.038373:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432416.038379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624960:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432416.038397:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432416.038407:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432416/real 1789432416] req@ffff9811d0bdd180 x1876350913624960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432416.038417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913624960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432416.038425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432416.038428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432416.038438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913624960:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432416.933972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432417.958020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432418.982007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432420.005972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432421.030017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432421.157944:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432421.157955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432421.157961:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432421.157970:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4790 00000100:00080000:2.0:1789432421.157979:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432421.158001:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432421.158034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913625088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432421.158059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432421.158063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432421.158066:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432421.158070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4790 00000100:00080000:2.0:1789432421.158074:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432421.158080:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432421.158086:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913625088:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789432421.158090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432421.158106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913625216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432421.158116:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432421.158119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432421.158121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432421.158125:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4790 00000100:00080000:2.0:1789432421.158128:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432421.158135:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789432421.158137:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432421.158142:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432421.158161:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913625344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432421.158172:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432421.158177:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1789432421.158178:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432421/real 1789432421] req@ffff9811d2102d80 x1876350913625088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432421.158180:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432421.158190:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4790 00000100:00080000:2.0:1789432421.158194:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789432421.158200:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913625088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432421.158201:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432421.158210:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789432421.158212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432421.158217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432421.158220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913625088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432421.158224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913625472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432421.158243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432421.158260:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432421.158268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432421/real 1789432421] req@ffff9811d2103800 x1876350913625216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432421.158275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913625216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432421.158279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432421.158281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432421.158283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432421.158286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432421.158295:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432421.158300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432421/real 1789432421] req@ffff9811d2100000 x1876350913625344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432421.158306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913625344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432421.158311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432421.158312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432421.158314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432421.158317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625472:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432421.158326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432421.158331:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432421/real 1789432421] req@ffff9811d2101500 x1876350913625472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432421.158338:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913625472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432421.158342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432421.158343:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432421.158345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625472:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432422.053934:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432423.078039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432424.101992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432425.125995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432426.149996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432426.277941:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432426.277969:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432426.277973:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432426.277984:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4795 00000100:00080000:2.0:1789432426.277995:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432426.278006:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432426.278045:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913625600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432426.278071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432426.278074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432426.278076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432426.278079:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4795 00000100:00080000:2.0:1789432426.278081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432426.278084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432426.278094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432426.278102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913625728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432426.278110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432426.278112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432426.278113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432426.278117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4795 00000100:00080000:2.0:1789432426.278120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432426.278126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432426.278127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913625600:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789432426.278130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432426.278144:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913625856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432426.278151:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432426.278155:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432426.278165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432426.278170:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4795 00000100:00080000:2.0:1789432426.278175:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432426.278193:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432426.278199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432426.278201:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432426.278218:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913625984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432426.278248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432426/real 1789432426] req@ffff9811d2103b80 x1876350913625600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432426.278279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913625600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432426.278290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432426.278294:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432426.278298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913625600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432426.278319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432426.278339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432426.278349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432426/real 1789432426] req@ffff9811d2101c00 x1876350913625728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432426.278360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913625728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432426.278368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432426.278371:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432426.278374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432426.278380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432426.278394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432426.278403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432426/real 1789432426] req@ffff9811d2102a00 x1876350913625856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432426.278420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913625856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432426.278427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432426.278431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432426.278434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432426.278440:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625984:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432426.278456:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432426.278465:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432426/real 1789432426] req@ffff9811d2100e00 x1876350913625984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432426.278476:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913625984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432426.278485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432426.278488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432426.278491:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913625984:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432427.174003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432428.198000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432429.222027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432430.246027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432431.270025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432431.397903:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432431.397923:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432431.397927:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432431.397936:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4800 00000100:00080000:2.0:1789432431.397943:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432431.397953:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432431.397982:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913626112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432431.398005:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432431.398008:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432431.398010:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432431.398014:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4800 00000100:00080000:2.0:1789432431.398017:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432431.398022:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432431.398031:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432431.398037:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913626112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432431.398044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913626240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432431.398054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432431.398058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432431.398059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432431.398063:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4800 00000100:00080000:2.0:1789432431.398066:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432431.398083:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432431.398088:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789432431.398092:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432431.398101:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913626368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432431.398111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432431.398114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432431.398116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432431.398120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4800 00000100:00080000:2.0:1789432431.398123:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789432431.398128:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432431/real 1789432431] req@ffff9811d2101f80 x1876350913626112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432431.398130:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432431.398138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432431.398153:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913626496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432431.398164:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913626112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432431.398173:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432431.398176:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432431.398179:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913626112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432431.398195:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432431.398210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432431.398217:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432431/real 1789432431] req@ffff9811d2101180 x1876350913626240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432431.398233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913626240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432431.398238:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432431.398240:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432431.398242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432431.398245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432431.398255:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432431.398261:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432431/real 1789432431] req@ffff9811d2101880 x1876350913626368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432431.398267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913626368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432431.398272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432431.398274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432431.398275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432431.398279:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626496:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432431.398287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432431.398292:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432431/real 1789432431] req@ffff9811d2100e00 x1876350913626496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432431.398298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913626496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432431.398307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432431.398316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432431.398318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432432.294023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432433.318029:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432434.341949:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432435.365997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432436.390010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432436.517947:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432436.517956:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432436.517962:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432436.517972:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4806 00000100:00080000:3.0:1789432436.517980:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432436.517991:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432436.518027:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913626624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432436.518051:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432436.518055:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432436.518056:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432436.518060:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4806 00000100:00080000:3.0:1789432436.518064:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432436.518068:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432436.518077:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432436.518095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913626752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432436.518097:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913626624:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432436.518114:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432436.518117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432436.518120:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432436.518124:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4806 00000100:00080000:3.0:1789432436.518127:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432436.518132:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432436.518138:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432436.518150:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913626880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432436.518161:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432436.518161:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432436.518167:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432436.518169:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432436.518175:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4806 00000100:00080000:3.0:1789432436.518182:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432436.518187:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432436.518193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432436.518200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432436/real 1789432436] req@ffff9811d1bc7100 x1876350913626624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432436.518207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913627008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432436.518227:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913626624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432436.518238:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432436.518242:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432436.518245:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913626624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432436.518264:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432436.518279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432436.518298:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432436/real 1789432436] req@ffff9811d1bc4700 x1876350913626752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432436.518305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913626752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432436.518311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432436.518313:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432436.518315:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432436.518319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432436.518329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432436.518335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432436/real 1789432436] req@ffff9811d1bc5500 x1876350913626880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432436.518341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913626880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432436.518346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432436.518347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432436.518349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913626880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432436.518352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627008:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432436.518361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432436.518366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432436/real 1789432436] req@ffff9811d1bc6680 x1876350913627008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432436.518372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913627008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432436.518377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432436.518378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432436.518380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432437.414026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432438.437995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432439.462146:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432440.485969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432441.510048:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432441.638092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432441.638102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432441.638106:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432441.638123:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4811 00000100:00080000:3.0:1789432441.638130:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432441.638141:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432441.638186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913627136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432441.638254:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432441.638258:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432441.638259:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432441.638264:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4811 00000100:00080000:3.0:1789432441.638268:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432441.638273:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432441.638282:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432441.638295:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913627264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432441.638303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913627136:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432441.638308:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432441.638313:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432441.638315:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432441.638320:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4811 00000100:00080000:3.0:1789432441.638324:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432441.638328:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432441.638333:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432441.638345:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913627392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432441.638356:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432441.638359:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432441.638361:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432441.638366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432441.638369:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4811 00000100:00080000:3.0:1789432441.638375:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432441.638381:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432441.638386:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432441.638398:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913627520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432441.638409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432441/real 1789432441] req@ffff9811d1bc6300 x1876350913627136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432441.638441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913627136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432441.638451:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432441.638455:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432441.638459:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913627136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432441.638480:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432441.638502:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432441.638515:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432441/real 1789432441] req@ffff9811d1bc5f80 x1876350913627264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432441.638526:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913627264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432441.638534:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432441.638537:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432441.638541:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432441.638547:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432441.638564:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432441.638574:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432441/real 1789432441] req@ffff9811d1bc4000 x1876350913627392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432441.638592:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913627392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432441.638600:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432441.638603:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432441.638606:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432441.638611:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627520:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432441.638630:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432441.638639:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432441/real 1789432441] req@ffff9811d1bc5880 x1876350913627520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432441.638649:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913627520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432441.638657:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432441.638659:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432441.638661:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432442.534012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432443.558039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432444.582053:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432445.606045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432446.630016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432446.757929:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432446.757937:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432446.757940:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432446.757947:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4816 00000100:00080000:2.0:1789432446.757955:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432446.757964:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432446.757992:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913627648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432446.758012:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432446.758015:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432446.758017:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432446.758020:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4816 00000100:00080000:2.0:1789432446.758023:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432446.758028:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432446.758037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432446.758048:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913627776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432446.758056:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432446.758059:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913627648:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432446.758059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432446.758061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432446.758064:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4816 00000100:00080000:2.0:1789432446.758066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432446.758071:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432446.758074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432446.758083:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913627904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432446.758091:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432446.758094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432446.758095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432446.758098:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4816 00000100:00080000:2.0:1789432446.758101:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432446.758105:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432446.758108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432446.758110:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432446.758116:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913628032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432446.758142:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432446/real 1789432446] req@ffff9811d2102a00 x1876350913627648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432446.758159:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913627648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432446.758171:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432446.758174:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432446.758176:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913627648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432446.758192:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432446.758208:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432446.758215:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432446/real 1789432446] req@ffff9811d2101c00 x1876350913627776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432446.758222:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913627776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432446.758227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432446.758228:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432446.758230:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432446.758233:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432446.758242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432446.758250:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432446/real 1789432446] req@ffff9811d2103b80 x1876350913627904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432446.758261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913627904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432446.758268:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432446.758271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432446.758274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913627904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432446.758280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628032:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432446.758300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432446.758311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432446/real 1789432446] req@ffff9811d2101500 x1876350913628032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432446.758322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913628032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432446.758330:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432446.758333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432446.758335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432447.653955:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432448.678062:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432449.702068:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432450.726069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432451.750006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432451.877948:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432451.877955:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432451.877958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432451.877966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4821 00000100:00080000:3.0:1789432451.877972:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432451.877981:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432451.878011:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913628160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432451.878031:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432451.878034:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432451.878035:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432451.878037:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4821 00000100:00080000:3.0:1789432451.878039:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432451.878042:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432451.878049:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432451.878059:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913628288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432451.878066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913628160:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432451.878071:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432451.878078:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432451.878080:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432451.878085:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4821 00000100:00080000:3.0:1789432451.878088:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432451.878092:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432451.878097:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432451.878109:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913628416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432451.878119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432451.878121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432451.878122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432451.878127:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432451.878129:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4821 00000100:00080000:3.0:1789432451.878134:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432451.878141:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432451.878146:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432451.878159:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913628544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432451.878168:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432451/real 1789432451] req@ffff9811d1bc7480 x1876350913628160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432451.878187:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913628160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432451.878197:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432451.878201:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432451.878205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913628160:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432451.878223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432451.878243:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432451.878253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432451/real 1789432451] req@ffff9811d1bc5c00 x1876350913628288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432451.878265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913628288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432451.878272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432451.878275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432451.878278:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432451.878284:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432451.878300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432451.878309:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432451/real 1789432451] req@ffff9811d1bc7800 x1876350913628416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432451.878319:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913628416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432451.878326:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432451.878328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432451.878331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432451.878336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628544:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432451.878351:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432451.878359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432451/real 1789432451] req@ffff9811d1bc5180 x1876350913628544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432451.878370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913628544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432451.878382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432451.878384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432451.878387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432452.773994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432453.798081:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432454.822035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432455.846080:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432456.869986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432456.997931:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432456.997940:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432456.997944:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432456.997953:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4826 00000100:00080000:3.0:1789432456.997962:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432456.997981:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432456.998016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913628672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432456.998046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432456.998048:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432456.998050:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432456.998054:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4826 00000100:00080000:3.0:1789432456.998057:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432456.998061:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432456.998069:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432456.998078:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913628672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432456.998082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913628800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432456.998091:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432456.998093:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432456.998095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432456.998097:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4826 00000100:00080000:3.0:1789432456.998100:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432456.998102:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432456.998104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432456.998113:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913628928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432456.998119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432456.998121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432456.998123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432456.998125:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4826 00000100:00080000:3.0:1789432456.998128:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432456.998130:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432456.998132:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432456.998138:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432456.998141:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913629056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432456.998177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432456/real 1789432456] req@ffff9811d1bc6a00 x1876350913628672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432456.998193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913628672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432456.998203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432456.998206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432456.998210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913628672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432456.998231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432456.998260:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432456.998270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432456/real 1789432456] req@ffff9811d1bc5180 x1876350913628800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432456.998277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913628800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432456.998283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432456.998285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432456.998287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432456.998290:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432456.998301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432456.998307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432456/real 1789432456] req@ffff9811d1bc7800 x1876350913628928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432456.998313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913628928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432456.998318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432456.998319:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432456.998321:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913628928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432456.998324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432456.998333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432456.998338:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432456/real 1789432456] req@ffff9811d1bc5c00 x1876350913629056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432456.998344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913629056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432456.998349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432456.998350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432456.998352:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432457.893986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432458.918007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432459.942027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432460.965963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432461.990004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432462.117928:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432462.117935:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432462.117939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432462.117946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4831 00000100:00080000:1.0:1789432462.117953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432462.117961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432462.117989:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913629184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432462.118015:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432462.118018:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432462.118019:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432462.118022:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4831 00000100:00080000:1.0:1789432462.118024:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432462.118027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432462.118034:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432462.118042:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913629312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432462.118049:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913629184:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432462.118051:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432462.118057:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432462.118058:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432462.118060:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4831 00000100:00080000:1.0:1789432462.118062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432462.118065:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432462.118068:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432462.118074:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913629440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432462.118086:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432462.118089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432462.118091:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432462.118094:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4831 00000100:00080000:1.0:1789432462.118098:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432462.118102:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432462.118108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432462.118125:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913629568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432462.118140:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432462.118175:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432462.118200:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432463.014032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432464.037992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432465.061956:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432465.190062:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432465.190093:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432465.190107:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432465.190140:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432465.190150:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432465.190158:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789432465.190185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432462/real 1789432465] req@ffff9811d0bddc00 x1876350913629312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432465.190207:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913629312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432465.190216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432465.190221:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432465.190225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432465.190232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432462/real 1789432465] req@ffff9811d0bdc380 x1876350913629440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432465.190243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913629440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432465.190256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432465.190259:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432465.190262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432465.190268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432462/real 1789432465] req@ffff9811d0bdd180 x1876350913629568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432465.190278:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913629568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432465.190286:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432465.190290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432465.190293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629568:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789432465.190299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432462/real 1789432465] req@ffff9811d0bdd500 x1876350913629184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432465.190309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913629184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432465.190317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432465.190321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432465.190324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913629184:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432466.086018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432467.109957:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432467.237952:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432467.237962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432467.237967:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432467.237977:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4836 00000100:00080000:1.0:1789432467.237986:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432467.237997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432467.238026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913629696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432467.238052:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432467.238055:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432467.238057:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432467.238062:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4836 00000100:00080000:1.0:1789432467.238067:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432467.238073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432467.238083:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432467.238094:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913629696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432467.238100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913629824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432467.238117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432467.238121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432467.238123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432467.238127:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4836 00000100:00080000:1.0:1789432467.238130:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432467.238134:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432467.238139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432467.238151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913629952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432467.238162:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432467.238166:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432467.238168:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432467.238171:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4836 00000100:00080000:1.0:1789432467.238175:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432467.238180:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432467.238186:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432467.238192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432467.238199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913630080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432467.238252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432467/real 1789432467] req@ffff9811d0bde680 x1876350913629696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432467.238275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913629696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432467.238288:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432467.238293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432467.238296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913629696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432467.238329:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432467.238356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432467.238368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432467/real 1789432467] req@ffff9811d0bdea00 x1876350913629824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432467.238381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913629824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432467.238399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432467.238403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432467.238406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432467.238412:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432467.238432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432467.238443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432467/real 1789432467] req@ffff9811d0bdfb80 x1876350913629952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432467.238453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913629952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432467.238466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432467.238469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432467.238472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913629952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432467.238479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432467.238497:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432467.238507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432467/real 1789432467] req@ffff9811d0bdf480 x1876350913630080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432467.238517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913630080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432467.238524:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432467.238526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432467.238529:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432468.134018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432469.157992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432470.182012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432471.206015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432472.230248:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432472.357955:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432472.357964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432472.357968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432472.357977:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4841 00000100:00080000:2.0:1789432472.357985:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432472.357996:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432472.358028:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913630208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432472.358054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432472.358058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432472.358060:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432472.358064:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4841 00000100:00080000:2.0:1789432472.358068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432472.358074:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432472.358084:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432472.358087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913630208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432472.358098:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913630336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432472.358109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432472.358113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432472.358115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432472.358119:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4841 00000100:00080000:2.0:1789432472.358130:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789432472.358134:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432472.358139:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432472.358145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432472.358158:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913630464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432472.358168:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432472.358172:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432472.358174:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789432472.358176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432472/real 1789432472] req@ffff9811d2100000 x1876350913630208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432472.358180:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4841 00000100:00080000:2.0:1789432472.358186:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432472.358193:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432472.358197:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913630208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789432472.358200:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789432472.358207:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432472.358210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432472.358213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913630208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432472.358217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913630592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432472.358240:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432472.358263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432472.358275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432472/real 1789432472] req@ffff9811d2103800 x1876350913630336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432472.358283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913630336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432472.358288:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432472.358290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432472.358292:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432472.358296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432472.358308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432472.358314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432472/real 1789432472] req@ffff9811d2102d80 x1876350913630464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432472.358320:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913630464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432472.358325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432472.358326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432472.358328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432472.358331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630592:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432472.358340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432472.358345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432472/real 1789432472] req@ffff9811d2101500 x1876350913630592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432472.358355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913630592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432472.358363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432472.358366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432472.358368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432473.254041:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432474.277992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432475.302004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432476.325975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432477.349962:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432477.477997:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432477.478009:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432477.478014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432477.478026:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4847 00000100:00080000:2.0:1789432477.478038:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432477.478053:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432477.478100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913630720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432477.478126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432477.478130:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432477.478131:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432477.478136:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4847 00000100:00080000:2.0:1789432477.478140:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432477.478145:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432477.478156:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432477.478174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913630848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432477.478175:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913630720:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432477.478199:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432477.478202:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432477.478204:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432477.478208:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4847 00000100:00080000:2.0:1789432477.478213:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432477.478218:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432477.478222:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432477.478233:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913630976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432477.478242:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432477.478249:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1789432477.478251:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432477.478252:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432477.478259:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4847 00000100:00080000:2.0:1789432477.478264:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432477.478271:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432477.478276:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432477.478289:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913631104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432477.478298:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432477/real 1789432477] req@ffff9811d2103b80 x1876350913630720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432477.478322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913630720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432477.478333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432477.478337:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432477.478341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913630720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432477.478362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432477.478381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432477.478390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432477/real 1789432477] req@ffff9811d2101c00 x1876350913630848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432477.478397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913630848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432477.478405:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432477.478407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432477.478409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432477.478413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432477.478427:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432477.478436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432477/real 1789432477] req@ffff9811d2102a00 x1876350913630976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432477.478446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913630976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432477.478454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432477.478457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432477.478460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913630976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432477.478466:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432477.478481:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432477.478487:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432477/real 1789432477] req@ffff9811d2100e00 x1876350913631104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432477.478494:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913631104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432477.478499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432477.478501:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432477.478503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432478.374002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432479.397980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432480.421982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432481.446012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432482.470062:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432482.597962:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432482.597972:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432482.597977:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432482.597988:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4852 00000100:00080000:1.0:1789432482.597997:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432482.598008:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432482.598051:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913631232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432482.598079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432482.598083:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432482.598085:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432482.598089:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4852 00000100:00080000:1.0:1789432482.598092:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432482.598099:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432482.598111:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432482.598116:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913631232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432482.598128:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913631360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432482.598140:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432482.598144:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432482.598146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432482.598150:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4852 00000100:00080000:1.0:1789432482.598155:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432482.598160:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432482.598166:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432482.598178:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913631488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432482.598192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432482.598195:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432482.598198:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432482.598199:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432482.598203:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4852 00000100:00080000:1.0:1789432482.598206:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432482.598210:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432482.598214:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432482.598224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913631616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432482.598254:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432482/real 1789432482] req@ffff9811d0bded80 x1876350913631232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432482.598282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913631232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432482.598291:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432482.598293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432482.598296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913631232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432482.598323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432482.598341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432482.598349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432482/real 1789432482] req@ffff9811d0bdc000 x1876350913631360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432482.598357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913631360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432482.598362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432482.598364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432482.598366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432482.598369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432482.598379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432482.598385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432482/real 1789432482] req@ffff9811d0bdc700 x1876350913631488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432482.598391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913631488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432482.598396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432482.598397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432482.598399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432482.598402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432482.598412:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432482.598417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432482/real 1789432482] req@ffff9811d0bdd880 x1876350913631616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432482.598424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913631616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432482.598429:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432482.598431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432482.598434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432483.494040:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432484.517994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432485.541972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432486.566000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432487.590016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432487.717942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432487.717950:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432487.717953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432487.717960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4857 00000100:00080000:2.0:1789432487.717967:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432487.717975:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432487.718007:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913631744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432487.718026:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432487.718029:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432487.718030:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432487.718033:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4857 00000100:00080000:2.0:1789432487.718035:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432487.718038:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432487.718046:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432487.718055:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913631744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432487.718056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913631872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432487.718069:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432487.718071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432487.718072:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432487.718073:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4857 00000100:00080000:2.0:1789432487.718075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432487.718077:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432487.718080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432487.718087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913632000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432487.718097:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432487.718100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432487.718102:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432487.718106:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4857 00000100:00080000:2.0:1789432487.718113:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432487.718119:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789432487.718124:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432487.718127:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432487.718140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913632128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432487.718166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432487/real 1789432487] req@ffff9811d2101880 x1876350913631744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432487.718184:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913631744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432487.718193:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432487.718195:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432487.718198:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913631744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432487.718213:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432487.718229:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432487.718237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432487/real 1789432487] req@ffff9811d2101180 x1876350913631872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432487.718244:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913631872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432487.718249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432487.718251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432487.718252:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913631872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432487.718256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432487.718264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432487.718269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432487/real 1789432487] req@ffff9811d2101f80 x1876350913632000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432487.718275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913632000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432487.718280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432487.718281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432487.718283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432487.718286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632128:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432487.718297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432487.718301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432487/real 1789432487] req@ffff9811d2100e00 x1876350913632128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432487.718308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913632128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432487.718312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432487.718314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432487.718316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432488.613964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432489.637971:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432490.662006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432491.685992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432492.709977:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432492.837936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432492.837943:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432492.837946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432492.837953:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4862 00000100:00080000:1.0:1789432492.837960:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432492.837969:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432492.838002:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913632256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432492.838026:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432492.838029:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432492.838031:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432492.838034:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4862 00000100:00080000:1.0:1789432492.838037:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432492.838040:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432492.838049:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432492.838062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913632384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432492.838072:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913632256:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432492.838075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432492.838082:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432492.838084:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432492.838088:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4862 00000100:00080000:1.0:1789432492.838091:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432492.838096:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432492.838101:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432492.838111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913632512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432492.838117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432492.838120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432492.838121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432492.838124:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4862 00000100:00080000:1.0:1789432492.838126:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432492.838128:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432492.838132:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432492.838137:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432492.838143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913632640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432492.838175:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432492/real 1789432492] req@ffff9811d0bdf100 x1876350913632256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432492.838192:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913632256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432492.838202:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432492.838206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432492.838210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913632256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432492.838229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432492.838252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432492.838263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432492/real 1789432492] req@ffff9811d0bde300 x1876350913632384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432492.838275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913632384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432492.838283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432492.838286:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432492.838289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432492.838294:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432492.838324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432492.838335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432492/real 1789432492] req@ffff9811d0bdf800 x1876350913632512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432492.838347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913632512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432492.838354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432492.838357:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432492.838368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432492.838375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632640:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432492.838394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432492.838405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432492/real 1789432492] req@ffff9811d0bdd880 x1876350913632640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432492.838416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913632640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432492.838439:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432492.838442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432492.838445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432493.734019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432494.757989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432495.781983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432496.805984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432497.830024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432497.957914:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432497.957921:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432497.957925:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432497.957933:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4867 00000100:00080000:1.0:1789432497.957939:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432497.957948:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432497.957974:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913632768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432497.957993:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432497.957995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432497.957997:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432497.958000:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4867 00000100:00080000:1.0:1789432497.958001:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432497.958003:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432497.958010:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432497.958018:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913632896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432497.958025:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432497.958029:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789432497.958030:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913632768:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432497.958032:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432497.958039:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4867 00000100:00080000:1.0:1789432497.958042:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432497.958047:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432497.958052:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432497.958065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913633024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432497.958081:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789432497.958084:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432497.958086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432497.958088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432497.958092:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4867 00000100:00080000:1.0:1789432497.958095:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432497.958099:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432497.958108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432497.958117:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432497/real 1789432497] req@ffff9811d0bdc700 x1876350913632768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432497.958119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913633152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432497.958153:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913632768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432497.958161:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432497.958164:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432497.958166:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913632768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432497.958182:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432497.958194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432497.958202:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432497/real 1789432497] req@ffff9811d0bdc000 x1876350913632896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432497.958208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913632896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432497.958213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432497.958215:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432497.958217:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913632896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432497.958221:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432497.958229:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432497.958234:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432497/real 1789432497] req@ffff9811d0bded80 x1876350913633024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432497.958240:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913633024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432497.958245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432497.958246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432497.958248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432497.958251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633152:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432497.958268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432497.958274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432497/real 1789432497] req@ffff9811d0bdf480 x1876350913633152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432497.958280:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913633152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432497.958285:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432497.958287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432497.958288:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432498.853981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432499.878093:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432500.902010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432501.926050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432502.950019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432503.077948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432503.077957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432503.077961:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432503.077969:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4872 00000100:00080000:2.0:1789432503.077976:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432503.077984:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432503.078010:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913633280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432503.078032:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432503.078035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432503.078037:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432503.078041:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4872 00000100:00080000:2.0:1789432503.078045:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432503.078050:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432503.078059:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432503.078070:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913633280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432503.078082:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913633408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432503.078093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432503.078096:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432503.078098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432503.078102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4872 00000100:00080000:2.0:1789432503.078117:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432503.078123:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432503.078128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789432503.078128:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432503.078143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913633536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432503.078153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789432503.078165:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432503/real 1789432503] req@ffff9811d2102a00 x1876350913633280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432503.078172:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432503.078176:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432503.078185:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4872 00000100:00080000:2.0:1789432503.078190:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789432503.078193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913633280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432503.078198:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432503.078204:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789432503.078214:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789432503.078219:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913633664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432503.078227:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432503.078232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913633280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432503.078254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432503.078273:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432503.078284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432503/real 1789432503] req@ffff9811d2101c00 x1876350913633408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432503.078303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913633408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432503.078311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432503.078314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432503.078317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432503.078322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432503.078343:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432503.078353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432503/real 1789432503] req@ffff9811d2103b80 x1876350913633536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432503.078363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913633536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432503.078376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432503.078379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432503.078382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432503.078399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432503.078417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432503.078426:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432503/real 1789432503] req@ffff9811d2101500 x1876350913633664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432503.078436:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913633664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432503.078444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432503.078447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432503.078450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432503.973992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432504.997989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432506.022013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432507.046061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432508.070066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432508.197936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432508.197946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432508.197950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432508.197960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4877 00000100:00080000:1.0:1789432508.197967:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432508.197976:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432508.198004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913633792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432508.198023:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432508.198025:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432508.198026:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432508.198029:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4877 00000100:00080000:1.0:1789432508.198032:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432508.198038:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432508.198048:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432508.198052:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913633792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432508.198062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913633920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432508.198075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432508.198078:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432508.198080:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432508.198084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4877 00000100:00080000:1.0:1789432508.198087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432508.198091:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432508.198095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432508.198103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913634048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432508.198110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432508.198113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432508.198114:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789432508.198118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432508.198120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4877 00000100:00080000:1.0:1789432508.198124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432508.198128:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432508.198132:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432508.198141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913634176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432508.198156:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432508/real 1789432508] req@ffff9811d0bdfb80 x1876350913633792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432508.198182:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913633792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432508.198192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432508.198196:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432508.198200:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913633792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432508.198219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432508.198238:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432508.198248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432508/real 1789432508] req@ffff9811d0bdea00 x1876350913633920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432508.198260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913633920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432508.198269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432508.198272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432508.198275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913633920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432508.198280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432508.198292:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432508.198300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432508/real 1789432508] req@ffff9811d0bde680 x1876350913634048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432508.198311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913634048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432508.198318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432508.198320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432508.198323:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432508.198328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432508.198341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432508.198361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432508/real 1789432508] req@ffff9811d0bdd500 x1876350913634176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432508.198373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913634176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432508.198382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432508.198384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432508.198387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432509.094023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432510.117988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432511.142116:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432512.166005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432513.189981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432513.318365:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432513.318375:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432513.318379:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432513.318388:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4882 00000100:00080000:0.0:1789432513.318397:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432513.318407:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432513.318441:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913634304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432513.318470:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432513.318474:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432513.318476:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432513.318480:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4882 00000100:00080000:0.0:1789432513.318483:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432513.318488:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432513.318497:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432513.318508:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913634432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432513.318518:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432513.318521:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432513.318522:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432513.318526:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4882 00000100:00080000:0.0:1789432513.318529:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432513.318533:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432513.318538:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432513.318549:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913634560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432513.318555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913634304:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432513.318566:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432513.318570:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432513.318571:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432513.318575:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4882 00000100:00080000:0.0:1789432513.318578:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432513.318582:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432513.318587:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432513.318599:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913634688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432513.318626:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432513.318663:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432513/real 1789432513] req@ffff9811c8cd5180 x1876350913634304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432513.318681:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913634304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432513.318692:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432513.318696:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432513.318700:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913634304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432513.318706:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432513.318737:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432513.318751:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432513/real 1789432513] req@ffff9811c8cd7800 x1876350913634432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432513.318764:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913634432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432513.318777:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432513.318780:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432513.318783:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432513.318804:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432513.318822:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432513.318853:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432513/real 1789432513] req@ffff9811c8cd4000 x1876350913634560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432513.318864:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913634560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432513.318871:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432513.318874:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432513.318877:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432513.318883:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634688:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432513.318900:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432513.318911:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432513/real 1789432513] req@ffff9811c8cd7100 x1876350913634688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432513.318921:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913634688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432513.318929:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432513.318932:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432513.318936:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432514.214012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432515.238103:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432516.262197:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432517.286037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432518.310025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432518.438004:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432518.438016:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432518.438020:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432518.438032:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4888 00000100:00080000:2.0:1789432518.438042:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432518.438054:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432518.438094:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913634816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432518.438121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432518.438125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432518.438127:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432518.438131:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4888 00000100:00080000:2.0:1789432518.438135:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432518.438143:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432518.438153:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432518.438176:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913634944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432518.438178:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913634816:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432518.438199:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432518.438203:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432518.438205:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432518.438210:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4888 00000100:00080000:2.0:1789432518.438214:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432518.438221:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432518.438227:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432518.438241:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913635072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432518.438252:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432518.438255:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432518.438257:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432518.438262:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4888 00000100:00080000:2.0:1789432518.438265:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432518.438270:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432518.438287:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432518.438289:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432518.438308:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913635200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432518.438350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432518/real 1789432518] req@ffff9811d2102d80 x1876350913634816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432518.438373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913634816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432518.438385:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432518.438389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432518.438393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913634816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432518.438420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432518.438445:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432518.438458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432518/real 1789432518] req@ffff9811d2103800 x1876350913634944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432518.438470:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913634944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432518.438479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432518.438482:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432518.438485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913634944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432518.438492:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432518.438509:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432518.438520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432518/real 1789432518] req@ffff9811d2100000 x1876350913635072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432518.438531:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913635072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432518.438539:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432518.438542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432518.438545:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432518.438552:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432518.438569:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432518.438579:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432518/real 1789432518] req@ffff9811d2101500 x1876350913635200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432518.438589:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913635200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432518.438597:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432518.438601:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432518.438604:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432519.333982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432520.357942:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432521.381982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432522.405996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432523.430003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432523.557990:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432523.557998:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432523.558001:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432523.558009:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4893 00000100:00080000:0.0:1789432523.558015:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432523.558024:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432523.558054:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913635328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432523.558073:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432523.558077:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432523.558079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432523.558083:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4893 00000100:00080000:0.0:1789432523.558087:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432523.558092:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432523.558101:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913635328:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789432523.558102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432523.558118:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913635456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432523.558129:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432523.558132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432523.558133:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432523.558136:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4893 00000100:00080000:0.0:1789432523.558139:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432523.558143:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432523.558146:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432523.558155:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913635584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432523.558162:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789432523.558164:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432523.558165:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432523.558168:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432523.558171:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4893 00000100:00080000:0.0:1789432523.558174:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432523.558177:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432523.558181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432523.558188:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913635712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432523.558208:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432523/real 1789432523] req@ffff9811c8cd6680 x1876350913635328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432523.558235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913635328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432523.558256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432523.558260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432523.558264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913635328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432523.558292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432523.558312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432523.558322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432523/real 1789432523] req@ffff9811c8cd4700 x1876350913635456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432523.558333:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913635456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432523.558340:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432523.558343:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432523.558346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432523.558352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432523.558379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432523.558390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432523/real 1789432523] req@ffff9811c8cd5c00 x1876350913635584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432523.558399:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913635584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432523.558406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432523.558409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432523.558420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432523.558427:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432523.558442:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432523.558450:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432523/real 1789432523] req@ffff9811c8cd7100 x1876350913635712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432523.558460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913635712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432523.558468:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432523.558471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432523.558474:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432524.454005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432525.478001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432526.502023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432527.525954:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432528.550024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432528.677917:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432528.677927:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432528.677930:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432528.677939:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4898 00000100:00080000:0.0:1789432528.677947:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432528.677958:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432528.677988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913635840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432528.678011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432528.678014:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432528.678016:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432528.678020:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4898 00000100:00080000:0.0:1789432528.678022:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432528.678027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432528.678036:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432528.678044:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913635840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432528.678050:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913635968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432528.678062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432528.678065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432528.678067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432528.678071:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4898 00000100:00080000:0.0:1789432528.678075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432528.678080:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432528.678085:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432528.678097:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913636096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432528.678106:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432528.678109:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432528.678111:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432528.678114:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4898 00000100:00080000:0.0:1789432528.678117:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432528.678121:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432528.678126:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432528.678137:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432528.678139:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913636224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432528.678165:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432528.678186:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432529.573996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432530.598078:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432531.621972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432531.750046:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432531.750094:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432531.750110:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432531.750146:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432531.750156:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432531.750163:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789432531.750416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432528/real 1789432531] req@ffff9811c8cd7480 x1876350913636224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432531.750440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913636224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432531.750453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432531.750458:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432531.750462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636224:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789432531.750470:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432528/real 1789432531] req@ffff9811c8cd7800 x1876350913635968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432531.750482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913635968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432531.750489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432531.750493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432531.750509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913635968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432531.750520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432528/real 1789432531] req@ffff9811c8cd5180 x1876350913636096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432531.750536:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913636096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432531.750543:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432531.750546:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432531.750548:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432531.750553:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432528/real 1789432531] req@ffff9811c8cd4000 x1876350913635840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432531.750564:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913635840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432531.750571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432531.750574:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432531.750577:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913635840:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432532.645975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432533.670008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432533.797954:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432533.797964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432533.797968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432533.797978:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4903 00000100:00080000:3.0:1789432533.797986:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432533.797996:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432533.798020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913636352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432533.798042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432533.798045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432533.798046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432533.798049:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4903 00000100:00080000:3.0:1789432533.798052:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432533.798054:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432533.798064:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432533.798074:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913636480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432533.798083:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432533.798086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432533.798088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432533.798092:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4903 00000100:00080000:3.0:1789432533.798094:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432533.798098:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432533.798103:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432533.798112:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913636608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432533.798121:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432533.798124:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432533.798126:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432533.798129:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4903 00000100:00080000:3.0:1789432533.798132:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432533.798137:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432533.798141:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432533.798151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913636736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432533.798746:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913636352:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789432533.798806:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432533.798864:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432533/real 1789432533] req@ffff9811d1bc7480 x1876350913636352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432533.798882:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913636352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432533.798892:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432533.798896:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432533.798899:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913636352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432533.798907:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432533.798931:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432533.798944:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432533/real 1789432533] req@ffff9811d1bc5880 x1876350913636480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432533.798956:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913636480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432533.798963:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432533.798967:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432533.798970:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432533.798979:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432533.798999:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432533.799010:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432533/real 1789432533] req@ffff9811d1bc4000 x1876350913636608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432533.799022:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913636608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432533.799030:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432533.799034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432533.799037:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432533.799043:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432533.799060:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432533.799067:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432533/real 1789432533] req@ffff9811d1bc5f80 x1876350913636736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432533.799075:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913636736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432533.799083:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432533.799086:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432533.799089:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432534.693973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432535.718055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432536.741975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432537.766050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432538.790029:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432538.917981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432538.917988:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432538.917992:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432538.918001:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4908 00000100:00080000:0.0:1789432538.918010:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432538.918020:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432538.918052:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913636864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432538.918075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432538.918078:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432538.918079:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432538.918082:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4908 00000100:00080000:0.0:1789432538.918084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432538.918088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432538.918095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432538.918102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913636992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432538.918108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432538.918110:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432538.918111:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432538.918113:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4908 00000100:00080000:0.0:1789432538.918115:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432538.918118:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432538.918121:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432538.918147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913637120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432538.918148:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913636864:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432538.918175:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432538.918177:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432538.918178:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432538.918180:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4908 00000100:00080000:0.0:1789432538.918182:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432538.918186:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432538.918189:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432538.918199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913637248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432538.918248:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432538.918293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432538/real 1789432538] req@ffff9811c8cd6300 x1876350913636864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432538.918309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913636864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432538.918320:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432538.918325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432538.918329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913636864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432538.918337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432538.918360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432538.918373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432538/real 1789432538] req@ffff9811c8cd7b80 x1876350913636992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432538.918385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913636992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432538.918393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432538.918396:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432538.918399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913636992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432538.918425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432538.918440:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432538.918447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432538/real 1789432538] req@ffff9811c8cd6a00 x1876350913637120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432538.918455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913637120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432538.918464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432538.918466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432538.918468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432538.918472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637248:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432538.918482:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432538.918487:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432538/real 1789432538] req@ffff9811c8cd4a80 x1876350913637248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432538.918494:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913637248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432538.918499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432538.918500:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432538.918502:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432539.814050:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432540.838051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432541.862098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432542.885983:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432543.910010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432544.037924:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432544.037932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432544.037935:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432544.037943:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4913 00000100:00080000:2.0:1789432544.037955:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432544.037965:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432544.037999:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913637376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432544.038021:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432544.038025:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432544.038027:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432544.038031:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4913 00000100:00080000:2.0:1789432544.038035:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432544.038040:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432544.038049:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432544.038063:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913637504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432544.038066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913637376:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432544.038082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432544.038085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432544.038087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432544.038103:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4913 00000100:00080000:2.0:1789432544.038107:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432544.038112:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432544.038118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789432544.038131:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432544.038137:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913637632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432544.038149:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432544.038152:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432544.038154:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432544.038159:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432544/real 1789432544] req@ffff9811d2103b80 x1876350913637376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789432544.038160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4913 00000100:00080000:2.0:1789432544.038167:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432544.038175:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432544.038178:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913637376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789432544.038182:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789432544.038187:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432544.038190:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432544.038192:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913637376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432544.038199:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913637760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432544.038211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432544.038227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432544.038235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432544/real 1789432544] req@ffff9811d2101c00 x1876350913637504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432544.038242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913637504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432544.038247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432544.038250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432544.038252:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432544.038255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432544.038263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432544.038267:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432544/real 1789432544] req@ffff9811d2102a00 x1876350913637632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432544.038274:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913637632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432544.038278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432544.038280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432544.038281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432544.038288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432544.038297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432544.038302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432544/real 1789432544] req@ffff9811d2100e00 x1876350913637760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432544.038308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913637760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432544.038313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432544.038315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432544.038317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913637760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432544.933946:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432545.957987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432546.981963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432548.006009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432549.029997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432549.157929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432549.157936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432549.157940:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432549.157947:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4918 00000100:00080000:0.0:1789432549.157953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432549.157961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432549.157988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913637888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432549.158010:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432549.158013:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432549.158014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432549.158017:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4918 00000100:00080000:0.0:1789432549.158019:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432549.158021:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432549.158028:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432549.158036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913638016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432549.158043:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432549.158045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432549.158046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432549.158048:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4918 00000100:00080000:0.0:1789432549.158049:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432549.158062:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432549.158066:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432549.158071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913637888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432549.158076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913638144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432549.158086:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432549.158088:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432549.158090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432549.158092:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4918 00000100:00080000:0.0:1789432549.158094:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432549.158097:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432549.158101:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432549.158109:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913638272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432549.158137:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432549.158174:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432549/real 1789432549] req@ffff9811c8cd5500 x1876350913637888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432549.158193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913637888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432549.158202:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432549.158206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432549.158209:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913637888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432549.158216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432549.158244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432549.158256:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432549/real 1789432549] req@ffff9811c8cd6d80 x1876350913638016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432549.158264:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913638016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432549.158273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432549.158275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432549.158277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432549.158293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432549.158305:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432549.158312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432549/real 1789432549] req@ffff9811c8cd5f80 x1876350913638144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432549.158318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913638144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432549.158323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432549.158325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432549.158326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432549.158330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432549.158345:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432549.158354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432549/real 1789432549] req@ffff9811c8cd4a80 x1876350913638272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432549.158365:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913638272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432549.158373:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432549.158376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432549.158380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432550.053994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432551.077987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432552.101987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432553.125996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432554.149986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432554.277943:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432554.277950:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432554.277953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432554.277964:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4923 00000100:00080000:3.0:1789432554.277971:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432554.277981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432554.278026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913638400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432554.278045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432554.278048:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432554.278049:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432554.278052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4923 00000100:00080000:3.0:1789432554.278055:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432554.278059:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432554.278066:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432554.278075:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913638528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432554.278083:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432554.278085:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432554.278086:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432554.278090:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913638400:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432554.278091:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4923 00000100:00080000:3.0:1789432554.278096:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432554.278099:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432554.278102:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432554.278110:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913638656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432554.278117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432554.278119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432554.278120:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432554.278122:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4923 00000100:00080000:3.0:1789432554.278124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432554.278127:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432554.278130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432554.278136:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913638784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432554.278147:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432554.278186:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432554/real 1789432554] req@ffff9811d1bc6300 x1876350913638400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432554.278203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913638400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432554.278213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432554.278217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432554.278220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913638400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432554.278235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432554.278251:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432554.278258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432554/real 1789432554] req@ffff9811d1bc6680 x1876350913638528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432554.278266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913638528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432554.278271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432554.278273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432554.278274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432554.278277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432554.278286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432554.278291:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432554/real 1789432554] req@ffff9811d1bc5500 x1876350913638656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432554.278297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913638656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432554.278301:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432554.278303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432554.278305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432554.278308:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638784:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432554.278316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432554.278321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432554/real 1789432554] req@ffff9811d1bc4700 x1876350913638784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432554.278327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913638784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432554.278331:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432554.278333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432554.278334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913638784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432555.173982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432556.197977:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432557.221993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432558.245941:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432559.270020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432559.397946:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432559.397955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432559.397959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432559.397976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4928 00000100:00080000:3.0:1789432559.397991:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432559.398000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432559.398031:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913638912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432559.398056:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432559.398060:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432559.398062:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432559.398066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4928 00000100:00080000:3.0:1789432559.398070:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432559.398075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432559.398084:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432559.398096:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913638912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432559.398101:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913639040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432559.398113:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432559.398116:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432559.398118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432559.398122:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4928 00000100:00080000:3.0:1789432559.398126:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432559.398130:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432559.398135:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432559.398147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913639168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432559.398157:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432559.398159:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432559.398163:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432559.398165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432559.398170:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4928 00000100:00080000:3.0:1789432559.398174:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432559.398180:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432559.398185:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432559.398197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913639296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432559.398199:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432559/real 1789432559] req@ffff9811d1bc7100 x1876350913638912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432559.398234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913638912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432559.398245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432559.398249:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432559.398253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913638912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432559.398273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432559.398294:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432559.398306:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432559/real 1789432559] req@ffff9811d1bc4700 x1876350913639040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432559.398317:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913639040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432559.398325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432559.398328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432559.398331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432559.398336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432559.398353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432559.398362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432559/real 1789432559] req@ffff9811d1bc5500 x1876350913639168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432559.398373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913639168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432559.398381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432559.398384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432559.398387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432559.398393:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432559.398410:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432559.398420:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432559/real 1789432559] req@ffff9811d1bc6680 x1876350913639296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432559.398431:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913639296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432559.398438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432559.398442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432559.398445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432560.294039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432561.317997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432562.341979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432563.366057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432564.389965:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432564.517916:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432564.517923:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432564.517926:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432564.517934:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4934 00000100:00080000:3.0:1789432564.517950:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432564.517960:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432564.517988:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913639424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432564.518008:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432564.518010:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432564.518011:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432564.518014:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4934 00000100:00080000:3.0:1789432564.518017:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432564.518021:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432564.518028:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432564.518033:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913639424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432564.518039:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913639552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432564.518047:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432564.518049:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432564.518050:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432564.518053:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4934 00000100:00080000:3.0:1789432564.518054:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432564.518057:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432564.518060:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432564.518067:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913639680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432564.518074:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432564.518076:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432564.518077:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432564.518080:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4934 00000800:00000100:1.0:1789432564.518083:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432564.518084:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432564.518089:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432564.518092:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432564.518098:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913639808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432564.518124:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432564/real 1789432564] req@ffff9811d1bc6300 x1876350913639424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432564.518137:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913639424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432564.518150:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432564.518154:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432564.518158:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913639424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432564.518176:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432564.518197:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432564.518207:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432564/real 1789432564] req@ffff9811d1bc5f80 x1876350913639552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432564.518214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913639552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432564.518220:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432564.518222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432564.518224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432564.518227:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432564.518238:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432564.518251:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432564/real 1789432564] req@ffff9811d1bc4000 x1876350913639680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432564.518258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913639680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432564.518263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432564.518264:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432564.518266:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432564.518269:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639808:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432564.518279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432564.518284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432564/real 1789432564] req@ffff9811d1bc5880 x1876350913639808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432564.518290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913639808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432564.518299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432564.518300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432564.518302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913639808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432565.414014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432566.437983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432567.461990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432568.485941:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432569.510076:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432569.637925:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432569.637935:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432569.637939:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432569.637949:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4939 00000100:00080000:3.0:1789432569.637958:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432569.637969:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432569.638002:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913639936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432569.638022:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432569.638025:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432569.638026:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432569.638029:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4939 00000100:00080000:3.0:1789432569.638031:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432569.638035:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432569.638042:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432569.638054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913639936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432569.638055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913640064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432569.638068:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432569.638070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432569.638071:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432569.638073:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4939 00000100:00080000:3.0:1789432569.638076:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432569.638078:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432569.638081:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432569.638090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913640192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432569.638099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432569.638111:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432569.638113:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432569.638118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432569.638121:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4939 00000100:00080000:3.0:1789432569.638126:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432569.638133:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432569.638138:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432569.638151:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913640320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432569.638160:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432569/real 1789432569] req@ffff9811d1bc7480 x1876350913639936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432569.638181:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913639936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432569.638191:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432569.638195:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432569.638198:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913639936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432569.638219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432569.638239:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432569.638251:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432569/real 1789432569] req@ffff9811d1bc5c00 x1876350913640064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432569.638262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913640064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432569.638270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432569.638273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432569.638276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432569.638281:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432569.638298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432569.638307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432569/real 1789432569] req@ffff9811d1bc7800 x1876350913640192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432569.638318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913640192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432569.638325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432569.638328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432569.638330:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432569.638336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432569.638352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432569.638362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432569/real 1789432569] req@ffff9811d1bc5180 x1876350913640320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432569.638371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913640320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432569.638379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432569.638382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432569.638385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432570.534057:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432571.558028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432572.582007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432573.606097:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432574.630022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432574.757919:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432574.757928:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432574.757932:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432574.757940:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4944 00000100:00080000:3.0:1789432574.757947:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432574.757955:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432574.757984:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913640448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432574.758005:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432574.758007:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432574.758009:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432574.758026:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4944 00000100:00080000:3.0:1789432574.758029:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432574.758033:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432574.758040:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432574.758046:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913640448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432574.758053:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913640576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432574.758060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432574.758062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432574.758063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432574.758065:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4944 00000100:00080000:3.0:1789432574.758067:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432574.758069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432574.758072:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432574.758080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913640704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432574.758086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432574.758088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432574.758090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432574.758091:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4944 00000100:00080000:3.0:1789432574.758094:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432574.758096:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432574.758099:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432574.758104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432574.758108:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913640832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432574.758144:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432574/real 1789432574] req@ffff9811d1bc6a00 x1876350913640448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432574.758156:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913640448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432574.758163:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432574.758165:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432574.758168:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913640448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432574.758184:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432574.758198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432574.758205:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432574/real 1789432574] req@ffff9811d1bc5180 x1876350913640576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432574.758211:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913640576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432574.758216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432574.758218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432574.758220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432574.758223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432574.758231:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432574.758236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432574/real 1789432574] req@ffff9811d1bc7800 x1876350913640704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432574.758242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913640704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432574.758250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432574.758252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432574.758254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432574.758257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432574.758265:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432574.758270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432574/real 1789432574] req@ffff9811d1bc5c00 x1876350913640832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432574.758276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913640832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432574.758281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432574.758282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432574.758284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913640832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432575.654011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432576.678017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432577.702009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432578.726070:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432579.750023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432579.877950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432579.877958:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432579.877961:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432579.877969:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4949 00000100:00080000:3.0:1789432579.877976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432579.877991:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432579.878025:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913640960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432579.878053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432579.878057:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432579.878058:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432579.878062:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4949 00000100:00080000:3.0:1789432579.878066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432579.878072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432579.878088:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913640960:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789432579.878091:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432579.878117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913641088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432579.878129:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432579.878132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432579.878134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432579.878138:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4949 00000100:00080000:3.0:1789432579.878142:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432579.878149:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432579.878150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789432579.878157:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432579.878176:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913641216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432579.878187:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432579.878191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432579/real 1789432579] req@ffff9811d1bc7480 x1876350913640960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432579.878192:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432579.878196:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432579.878203:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4949 00000100:00080000:3.0:1789432579.878209:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432579.878216:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432579.878221:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913640960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789432579.878224:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789432579.878235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789432579.878241:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913641344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432579.878243:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432579.878251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913640960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432579.878275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432579.878304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432579.878324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432579/real 1789432579] req@ffff9811d1bc5880 x1876350913641088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432579.878341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913641088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432579.878349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432579.878353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432579.878356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432579.878362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432579.878379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432579.878389:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432579/real 1789432579] req@ffff9811d1bc4000 x1876350913641216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432579.878399:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913641216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432579.878406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432579.878409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432579.878412:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432579.878417:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641344:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432579.878435:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432579.878445:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432579/real 1789432579] req@ffff9811d1bc5f80 x1876350913641344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432579.878457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913641344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432579.878465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432579.878467:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432579.878470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432580.773987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432581.797983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432582.822008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432583.846013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432584.870002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432584.997968:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432584.997983:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432584.997988:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432584.998001:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4954 00000100:00080000:3.0:1789432584.998019:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432584.998034:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432584.998080:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913641472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432584.998109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432584.998112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432584.998113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432584.998116:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4954 00000100:00080000:3.0:1789432584.998120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432584.998124:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432584.998133:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432584.998143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913641600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432584.998150:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432584.998153:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432584.998154:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432584.998156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4954 00000100:00080000:3.0:1789432584.998158:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432584.998161:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432584.998165:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432584.998166:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913641472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432584.998173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913641728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432584.998181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432584.998183:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432584.998184:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432584.998186:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4954 00000100:00080000:3.0:1789432584.998187:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432584.998190:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432584.998193:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432584.998199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913641856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432584.998247:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432584.998294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432584/real 1789432584] req@ffff9811d1bc6300 x1876350913641472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432584.998312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913641472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432584.998323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432584.998327:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432584.998331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913641472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432584.998339:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432584.998361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432584.998374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432584/real 1789432584] req@ffff9811d1bc6680 x1876350913641600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432584.998386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913641600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432584.998400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432584.998404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432584.998407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432584.998446:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432584.998466:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432584.998478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432584/real 1789432584] req@ffff9811d1bc5500 x1876350913641728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432584.998489:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913641728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432584.998497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432584.998501:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432584.998504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432584.998510:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432584.998529:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432584.998540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432584/real 1789432584] req@ffff9811d1bc4700 x1876350913641856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432584.998550:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913641856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432584.998558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432584.998561:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432584.998565:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913641856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432585.893952:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432586.918037:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432587.941986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432588.966013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432589.990005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432590.117953:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432590.117964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432590.117969:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432590.117979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4959 00000100:00080000:3.0:1789432590.117988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432590.118012:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432590.118048:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913641984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432590.118073:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432590.118077:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432590.118079:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432590.118083:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4959 00000100:00080000:3.0:1789432590.118087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432590.118092:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432590.118100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432590.118114:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913642112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432590.118124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432590.118127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432590.118128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432590.118131:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4959 00000100:00080000:3.0:1789432590.118134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432590.118139:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432590.118144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432590.118155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913642240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432590.118166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432590.118167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913641984:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432590.118173:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432590.118177:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432590.118182:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4959 00000100:00080000:3.0:1789432590.118185:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432590.118192:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432590.118197:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432590.118209:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913642368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432590.118230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432590.118271:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432590/real 1789432590] req@ffff9811d1bc7100 x1876350913641984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432590.118289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913641984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432590.118300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432590.118304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432590.118307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913641984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432590.118315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432590.118334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432590.118344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432590/real 1789432590] req@ffff9811d1bc4700 x1876350913642112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432590.118355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913642112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432590.118363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432590.118366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432590.118369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432590.118388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432590.118404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432590.118414:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432590/real 1789432590] req@ffff9811d1bc5500 x1876350913642240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432590.118424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913642240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432590.118432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432590.118436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432590.118439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432590.118444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432590.118462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432590.118471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432590/real 1789432590] req@ffff9811d1bc6680 x1876350913642368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432590.118481:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913642368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432590.118489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432590.118491:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432590.118494:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432591.014037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432592.038010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432593.062420:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432594.085990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432595.110014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432595.237915:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432595.237922:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432595.237926:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432595.237933:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4964 00000100:00080000:3.0:1789432595.237939:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432595.237947:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432595.237974:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913642496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432595.237992:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432595.237994:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432595.237995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432595.237998:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4964 00000100:00080000:3.0:1789432595.238000:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432595.238003:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432595.238010:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432595.238019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913642624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432595.238024:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913642496:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432595.238028:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432595.238031:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432595.238032:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432595.238035:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4964 00000100:00080000:3.0:1789432595.238037:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432595.238041:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432595.238045:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432595.238057:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913642752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432595.238067:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432595.238071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432595.238073:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432595.238076:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4964 00000100:00080000:3.0:1789432595.238080:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432595.238090:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432595.238096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432595.238100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432595.238111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913642880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432595.238123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432595.238148:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432596.134023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432597.157986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432598.181997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432598.310080:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432598.310112:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432598.310126:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432598.310162:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432598.310170:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432598.310180:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432598.310201:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432595/real 1789432598] req@ffff9811d1bc5880 x1876350913642880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432598.310225:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913642880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432598.310235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432598.310238:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432598.310241:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642880:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432598.310247:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432595/real 1789432598] req@ffff9811d1bc5f80 x1876350913642624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432598.310258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913642624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432598.310266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432598.310269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432598.310273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432598.310279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432595/real 1789432598] req@ffff9811d1bc4000 x1876350913642752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432598.310290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913642752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432598.310298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432598.310302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432598.310305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913642752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432598.310313:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432595/real 1789432598] req@ffff9811d1bc6300 x1876350913642496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432598.310326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913642496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432598.310334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432598.310337:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432598.310340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913642496:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432599.205993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432600.230018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432600.357936:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432600.357943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432600.357946:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432600.357954:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4969 00000100:00080000:3.0:1789432600.357962:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432600.357973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432600.358006:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913643008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432600.358028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432600.358032:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432600.358034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432600.358037:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4969 00000100:00080000:3.0:1789432600.358041:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432600.358047:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432600.358056:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432600.358064:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913643008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432600.358072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913643136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432600.358084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432600.358088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432600.358090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432600.358093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4969 00000100:00080000:3.0:1789432600.358097:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432600.358102:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432600.358108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432600.358119:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913643264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432600.358131:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789432600.358136:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432600.358137:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432600.358139:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432600.358147:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4969 00000100:00080000:3.0:1789432600.358152:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432600.358159:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432600.358164:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432600.358176:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913643392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432600.358184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432600/real 1789432600] req@ffff9811d1bc7480 x1876350913643008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432600.358204:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913643008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432600.358215:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432600.358219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432600.358222:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913643008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432600.358244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432600.358266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432600.358278:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432600/real 1789432600] req@ffff9811d1bc5c00 x1876350913643136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432600.358289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913643136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432600.358297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432600.358300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432600.358303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432600.358310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432600.358327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432600.358338:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432600/real 1789432600] req@ffff9811d1bc7800 x1876350913643264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432600.358349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913643264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432600.358356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432600.358359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432600.358362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432600.358368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432600.358387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432600.358397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432600/real 1789432600] req@ffff9811d1bc5180 x1876350913643392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432600.358408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913643392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432600.358416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432600.358419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432600.358423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432601.253989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432602.278000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432603.302007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432604.325982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432605.349999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432605.478026:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432605.478033:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432605.478036:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432605.478044:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4975 00000100:00080000:2.0:1789432605.478050:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432605.478058:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432605.478086:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913643520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432605.478103:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432605.478106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432605.478107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432605.478110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4975 00000100:00080000:2.0:1789432605.478111:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432605.478114:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432605.478120:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432605.478128:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913643648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432605.478135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432605.478137:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432605.478138:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432605.478141:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4975 00000100:00080000:2.0:1789432605.478143:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789432605.478147:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913643520:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432605.478148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432605.478153:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432605.478167:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913643776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432605.478183:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432605.478187:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432605.478193:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432605.478197:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4975 00000100:00080000:2.0:1789432605.478201:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432605.478207:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432605.478208:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432605.478216:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432605.478229:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913643904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432605.478258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432605/real 1789432605] req@ffff9811d2101f80 x1876350913643520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432605.478272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913643520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432605.478280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432605.478283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432605.478285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913643520:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432605.478303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432605.478323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432605.478335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432605/real 1789432605] req@ffff9811d2101180 x1876350913643648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432605.478346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913643648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432605.478351:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432605.478353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432605.478355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432605.478358:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432605.478370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432605.478377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432605/real 1789432605] req@ffff9811d2101880 x1876350913643776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432605.478383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913643776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432605.478387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432605.478389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432605.478391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432605.478394:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643904:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432605.478402:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432605.478407:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432605/real 1789432605] req@ffff9811d2100e00 x1876350913643904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432605.478413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913643904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432605.478417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432605.478419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432605.478420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913643904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432606.374049:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432607.398008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432608.422055:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432609.446037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432610.469997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432610.597956:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432610.597965:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432610.597969:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432610.597979:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4980 00000100:00080000:3.0:1789432610.597987:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432610.597999:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432610.598033:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913644032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432610.598055:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432610.598059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432610.598060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432610.598064:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4980 00000100:00080000:3.0:1789432610.598068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432610.598073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432610.598082:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432610.598091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913644032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432610.598097:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913644160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432610.598109:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432610.598112:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432610.598114:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432610.598118:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4980 00000100:00080000:3.0:1789432610.598121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432610.598126:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432610.598130:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432610.598140:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913644288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432610.598149:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789432610.598153:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432610.598154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432610.598156:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432610.598160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4980 00000100:00080000:3.0:1789432610.598163:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432610.598169:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432610.598174:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432610.598184:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913644416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432610.598202:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432610/real 1789432610] req@ffff9811d1bc6a00 x1876350913644032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432610.598228:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913644032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432610.598239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432610.598242:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432610.598246:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913644032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432610.598265:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432610.598286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432610.598299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432610/real 1789432610] req@ffff9811d1bc5180 x1876350913644160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432610.598321:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913644160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432610.598329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432610.598333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432610.598336:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432610.598341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432610.598359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432610.598370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432610/real 1789432610] req@ffff9811d1bc7800 x1876350913644288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432610.598381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913644288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432610.598388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432610.598391:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432610.598394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432610.598399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432610.598418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432610.598428:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432610/real 1789432610] req@ffff9811d1bc5c00 x1876350913644416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432610.598440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913644416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432610.598447:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432610.598450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432610.598453:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432611.493997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432612.517997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432613.541992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432614.565973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432615.590061:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432615.718093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432615.718100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432615.718103:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432615.718110:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4985 00000100:00080000:2.0:1789432615.718117:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432615.718126:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432615.718156:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913644544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432615.718178:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432615.718180:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432615.718182:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432615.718186:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4985 00000100:00080000:2.0:1789432615.718190:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432615.718194:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432615.718203:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432615.718218:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913644672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432615.718219:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913644544:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432615.718236:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432615.718238:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432615.718240:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432615.718243:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4985 00000100:00080000:2.0:1789432615.718245:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432615.718249:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432615.718261:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432615.718274:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913644800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432615.718283:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432615.718286:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432615.718290:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432615.718293:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432615.718297:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4985 00000100:00080000:2.0:1789432615.718302:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432615.718308:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432615.718312:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432615.718322:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913644928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432615.718337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432615/real 1789432615] req@ffff9811d2102a00 x1876350913644544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432615.718358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913644544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432615.718365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432615.718368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432615.718371:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913644544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432615.718388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432615.718404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432615.718425:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432615/real 1789432615] req@ffff9811d2101c00 x1876350913644672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432615.718438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913644672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432615.718456:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432615.718459:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432615.718462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432615.718468:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432615.718486:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432615.718494:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432615/real 1789432615] req@ffff9811d2103b80 x1876350913644800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432615.718501:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913644800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432615.718505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432615.718507:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432615.718517:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432615.718522:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432615.718546:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432615.718556:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432615/real 1789432615] req@ffff9811d2101500 x1876350913644928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432615.718567:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913644928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432615.718589:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432615.718591:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432615.718593:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913644928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432616.613985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432617.637978:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432618.661997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432619.686014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432620.710024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432620.837952:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432620.837961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432620.837965:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432620.837974:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4990 00000100:00080000:0.0:1789432620.837983:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432620.837992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432620.838026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913645056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432620.838050:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432620.838053:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432620.838055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432620.838059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4990 00000100:00080000:0.0:1789432620.838062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432620.838068:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432620.838078:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432620.838084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913645056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432620.838093:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913645184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432620.838103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432620.838106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432620.838108:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432620.838111:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4990 00000100:00080000:0.0:1789432620.838122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432620.838126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432620.838131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789432620.838136:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432620.838145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913645312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432620.838157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432620.838160:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432620.838162:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432620.838166:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4990 00000100:00080000:0.0:1789432620.838170:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432620.838177:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432620.838179:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432620/real 1789432620] req@ffff9811c8cd6a00 x1876350913645056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789432620.838183:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432620.838198:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913645440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432620.838202:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913645056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432620.838225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432620.838230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432620.838235:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913645056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432620.838268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432620.838290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432620.838302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432620/real 1789432620] req@ffff9811c8cd7b80 x1876350913645184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432620.838313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913645184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432620.838321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432620.838324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432620.838326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432620.838332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432620.838349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432620.838359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432620/real 1789432620] req@ffff9811c8cd6300 x1876350913645312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432620.838368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913645312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432620.838375:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432620.838378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432620.838381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432620.838386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645440:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432620.838404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432620.838415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432620/real 1789432620] req@ffff9811c8cd4000 x1876350913645440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432620.838426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913645440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432620.838433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432620.838436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432620.838438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432621.733985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432622.757964:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432623.781984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432624.805954:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432625.830034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432625.957950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432625.957961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432625.957966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432625.957976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 4995 00000100:00080000:1.0:1789432625.957984:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432625.957992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432625.958020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913645568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432625.958041:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432625.958044:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432625.958045:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432625.958048:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4995 00000100:00080000:1.0:1789432625.958050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432625.958052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432625.958059:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432625.958070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913645696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432625.958077:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432625.958079:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432625.958080:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432625.958082:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4995 00000100:00080000:1.0:1789432625.958084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432625.958087:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432625.958090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432625.958100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913645568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432625.958103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913645824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432625.958131:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432625.958134:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432625.958136:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432625.958140:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 4995 00000100:00080000:1.0:1789432625.958144:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432625.958148:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432625.958153:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432625.958166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913645952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789432625.958206:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432625.958255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432625/real 1789432625] req@ffff9811d0bdd180 x1876350913645568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432625.958286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913645568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432625.958298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432625.958303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432625.958307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913645568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432625.958315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432625.958338:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432625.958349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432625/real 1789432625] req@ffff9811d0bdc380 x1876350913645696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432625.958361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913645696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432625.958370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432625.958374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432625.958377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432625.958403:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432625.958420:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432625.958431:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432625/real 1789432625] req@ffff9811d0bddc00 x1876350913645824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432625.958442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913645824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432625.958450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432625.958453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432625.958457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432625.958463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432625.958480:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432625.958490:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432625/real 1789432625] req@ffff9811d0bdd500 x1876350913645952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432625.958516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913645952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432625.958525:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432625.958529:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432625.958532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913645952:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432626.854028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432627.878157:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432628.902038:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432629.925992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432630.950028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432631.077956:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432631.077965:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432631.077970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432631.077979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5000 00000100:00080000:2.0:1789432631.077987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432631.077997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432631.078031:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913646080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432631.078057:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432631.078061:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432631.078062:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432631.078066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5000 00000100:00080000:2.0:1789432631.078068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432631.078072:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432631.078079:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432631.078090:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913646208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432631.078101:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789432631.078102:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913646080:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432631.078107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432631.078112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432631.078116:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5000 00000100:00080000:2.0:1789432631.078120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432631.078126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432631.078131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432631.078143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913646336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432631.078153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432631.078158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1789432631.078159:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432631.078161:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432631.078168:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5000 00000100:00080000:2.0:1789432631.078172:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432631.078177:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432631.078181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432631.078192:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913646464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432631.078207:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432631/real 1789432631] req@ffff9811d2100000 x1876350913646080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432631.078226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913646080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432631.078237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432631.078241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432631.078245:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913646080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432631.078263:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432631.078288:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432631.078297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432631/real 1789432631] req@ffff9811d2103800 x1876350913646208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432631.078305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913646208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432631.078310:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432631.078312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432631.078314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432631.078317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432631.078327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432631.078332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432631/real 1789432631] req@ffff9811d2102d80 x1876350913646336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432631.078339:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913646336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432631.078343:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432631.078345:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432631.078346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432631.078350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646464:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432631.078360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432631.078365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432631/real 1789432631] req@ffff9811d2101500 x1876350913646464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432631.078371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913646464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432631.078376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432631.078377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432631.078379:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646464:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432631.974024:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432632.998065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432634.022007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432635.045992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432636.070008:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432636.197928:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432636.197935:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432636.197938:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432636.197946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5005 00000100:00080000:1.0:1789432636.197953:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432636.197961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432636.197989:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913646592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432636.198005:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432636.198008:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432636.198009:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432636.198011:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5005 00000100:00080000:1.0:1789432636.198013:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432636.198016:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432636.198023:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432636.198037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913646720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432636.198046:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432636.198049:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432636.198051:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432636.198054:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5005 00000100:00080000:1.0:1789432636.198057:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432636.198061:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432636.198065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432636.198075:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913646848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432636.198084:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432636.198087:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432636.198088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432636.198091:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5005 00000100:00080000:1.0:1789432636.198094:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432636.198098:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432636.198103:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432636.198111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913646976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432636.198407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913646592:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789432636.198472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432636.198507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432636/real 1789432636] req@ffff9811d0bde680 x1876350913646592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432636.198523:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913646592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432636.198534:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432636.198537:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432636.198541:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913646592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432636.198548:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432636.198569:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432636.198581:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432636/real 1789432636] req@ffff9811d0bdea00 x1876350913646720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432636.198594:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913646720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432636.198602:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432636.198605:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432636.198608:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432636.198613:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432636.198629:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432636.198636:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432636/real 1789432636] req@ffff9811d0bdfb80 x1876350913646848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432636.198642:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913646848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432636.198647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432636.198649:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432636.198651:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432636.198654:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646976:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432636.198670:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432636.198676:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432636/real 1789432636] req@ffff9811d0bdf480 x1876350913646976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432636.198682:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913646976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432636.198691:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432636.198693:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432636.198695:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913646976:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432637.094020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432638.118071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432639.142006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432640.165998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432641.190015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432641.317944:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432641.317951:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432641.317955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432641.317962:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5010 00000100:00080000:0.0:1789432641.317968:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432641.317977:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432641.318005:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913647104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432641.318026:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432641.318029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432641.318030:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432641.318033:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5010 00000100:00080000:0.0:1789432641.318034:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432641.318037:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432641.318043:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432641.318060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913647232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432641.318071:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789432641.318075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913647104:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432641.318076:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432641.318079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432641.318088:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5010 00000100:00080000:0.0:1789432641.318092:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432641.318096:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432641.318101:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432641.318113:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913647360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432641.318124:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432641.318128:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432641.318130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432641.318134:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5010 00000100:00080000:0.0:1789432641.318138:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789432641.318142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432641.318146:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432641.318152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432641.318164:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913647488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432641.318188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432641/real 1789432641] req@ffff9811c8cd5180 x1876350913647104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432641.318206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913647104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432641.318226:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432641.318230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432641.318233:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913647104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432641.318254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432641.318276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432641.318289:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432641/real 1789432641] req@ffff9811c8cd7800 x1876350913647232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432641.318300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913647232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432641.318308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432641.318311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432641.318314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432641.318320:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432641.318342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432641.318354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432641/real 1789432641] req@ffff9811c8cd7480 x1876350913647360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432641.318364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913647360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432641.318371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432641.318374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432641.318377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432641.318382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647488:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432641.318417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432641.318427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432641/real 1789432641] req@ffff9811c8cd7100 x1876350913647488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432641.318437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913647488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432641.318445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432641.318447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432641.318450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647488:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432642.213991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432643.238063:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432644.262032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432645.286079:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432646.310008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432646.437917:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432646.437927:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432646.437932:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432646.437942:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5016 00000100:00080000:0.0:1789432646.437950:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432646.437961:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432646.437989:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913647616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432646.438008:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432646.438010:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432646.438012:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432646.438014:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5016 00000100:00080000:0.0:1789432646.438016:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432646.438020:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432646.438027:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432646.438035:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913647744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432646.438042:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432646.438045:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432646.438046:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789432646.438048:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913647616:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432646.438049:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5016 00000100:00080000:0.0:1789432646.438054:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432646.438057:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432646.438060:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432646.438068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913647872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432646.438075:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432646.438077:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432646.438079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432646.438082:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5016 00000100:00080000:0.0:1789432646.438086:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432646.438091:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432646.438096:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789432646.438101:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432646.438110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913648000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789432646.438133:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432646/real 1789432646] req@ffff9811c8cd5c00 x1876350913647616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432646.438150:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913647616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432646.438160:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432646.438164:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432646.438168:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913647616:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432646.438189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432646.438211:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432646.438222:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432646/real 1789432646] req@ffff9811c8cd4700 x1876350913647744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432646.438234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913647744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432646.438242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432646.438246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432646.438249:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432646.438254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789432646.438270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432646.438280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432646/real 1789432646] req@ffff9811c8cd6680 x1876350913647872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432646.438291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913647872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432646.438305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432646.438309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432646.438312:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913647872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432646.438318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648000:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789432646.438336:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432646.438346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432646/real 1789432646] req@ffff9811c8cd7100 x1876350913648000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432646.438370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913648000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432646.438379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789432646.438382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789432646.438385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648000:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432647.334032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432648.357996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432649.381984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432650.405963:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432651.430009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432651.557946:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432651.557956:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432651.557959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432651.557970:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5021 00000100:00080000:3.0:1789432651.557978:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432651.557989:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432651.558020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913648128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432651.558042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432651.558046:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432651.558048:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432651.558052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5021 00000100:00080000:3.0:1789432651.558056:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432651.558061:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432651.558071:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432651.558084:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913648256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432651.558094:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432651.558098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432651.558100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432651.558105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5021 00000100:00080000:3.0:1789432651.558108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432651.558114:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432651.558118:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432651.558132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913648384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432651.558133:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913648128:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432651.558154:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432651.558157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432651.558159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432651.558164:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5021 00000100:00080000:3.0:1789432651.558167:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432651.558174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432651.558180:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432651.558192:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913648512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432651.558241:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432651.558304:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432651/real 1789432651] req@ffff9811d1bc7480 x1876350913648128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432651.558342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913648128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432651.558354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432651.558359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432651.558363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913648128:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432651.558372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432651.558395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432651.558408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432651/real 1789432651] req@ffff9811d1bc6300 x1876350913648256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432651.558420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913648256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432651.558428:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432651.558431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432651.558443:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432651.558473:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432651.558493:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432651.558505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432651/real 1789432651] req@ffff9811d1bc4000 x1876350913648384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432651.558517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913648384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432651.558531:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432651.558534:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432651.558537:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432651.558543:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648512:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432651.558575:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432651.558587:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432651/real 1789432651] req@ffff9811d1bc5f80 x1876350913648512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432651.558598:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913648512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432651.558607:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432651.558611:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432651.558622:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648512:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432652.453988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432653.478055:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432654.501988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432655.525972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432656.550038:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432656.677917:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432656.677924:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432656.677927:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432656.677936:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5026 00000100:00080000:3.0:1789432656.677943:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432656.677951:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432656.677979:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913648640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432656.677998:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432656.678001:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432656.678002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432656.678005:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5026 00000100:00080000:3.0:1789432656.678006:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432656.678009:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432656.678015:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432656.678026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913648768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432656.678036:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432656.678040:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913648640:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432656.678041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432656.678045:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432656.678052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5026 00000100:00080000:3.0:1789432656.678056:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432656.678060:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432656.678065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432656.678077:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913648896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432656.678087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432656.678090:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432656.678092:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432656.678095:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5026 00000100:00080000:3.0:1789432656.678098:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432656.678104:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432656.678106:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789432656.678109:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432656.678119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913649024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432656.678146:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432656/real 1789432656] req@ffff9811d1bc5880 x1876350913648640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432656.678159:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913648640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432656.678165:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432656.678168:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432656.678171:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913648640:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432656.678190:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432656.678210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432656.678223:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432656/real 1789432656] req@ffff9811d1bc6680 x1876350913648768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432656.678234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913648768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432656.678242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432656.678245:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432656.678248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432656.678253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432656.678270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432656.678290:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432656/real 1789432656] req@ffff9811d1bc5500 x1876350913648896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432656.678298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913648896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432656.678302:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432656.678304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432656.678306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913648896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432656.678310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649024:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432656.678326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432656.678335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432656/real 1789432656] req@ffff9811d1bc4700 x1876350913649024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432656.678347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913649024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432656.678354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432656.678357:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432656.678360:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649024:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432657.574003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432658.598017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432659.621996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432660.646031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432661.669970:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432661.797917:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432661.797924:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432661.797928:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432661.797935:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5031 00000100:00080000:3.0:1789432661.797943:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432661.797952:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432661.797981:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913649152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432661.798001:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432661.798005:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432661.798007:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432661.798009:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5031 00000100:00080000:3.0:1789432661.798011:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432661.798014:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432661.798021:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432661.798030:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913649280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432661.798037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432661.798041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789432661.798043:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913649152:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432661.798043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432661.798049:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5031 00000100:00080000:3.0:1789432661.798053:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432661.798055:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432661.798059:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432661.798072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913649408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432661.798082:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432661.798085:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432661.798087:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432661.798091:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5031 00000100:00080000:3.0:1789432661.798095:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432661.798101:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432661.798106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432661.798118:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913649536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432661.798141:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432661.798169:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432661.798188:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649536:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432662.693982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432663.718035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432664.742237:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789432664.870142:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789432664.870184:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789432664.870197:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789432664.870255:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789432664.870266:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789432664.870285:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432664.870293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432661/real 1789432664] req@ffff9811d1bc4700 x1876350913649280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432664.870318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913649280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432664.870329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432664.870334:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432664.870339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432664.870347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432661/real 1789432664] req@ffff9811d1bc5500 x1876350913649408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432664.870358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913649408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432664.870365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432664.870369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432664.870372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432664.870378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432661/real 1789432664] req@ffff9811d1bc6680 x1876350913649536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432664.870387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913649536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432664.870394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432664.870397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432664.870400:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649536:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432664.870405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432661/real 1789432664] req@ffff9811d1bc7100 x1876350913649152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432664.870415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913649152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432664.870421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432664.870424:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432664.870427:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913649152:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432665.765992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432666.790003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432666.917949:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432666.917957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432666.917960:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432666.917992:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5036 00000100:00080000:3.0:1789432666.918002:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432666.918014:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432666.918047:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913649664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432666.918076:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432666.918080:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432666.918082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432666.918087:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5036 00000100:00080000:3.0:1789432666.918092:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432666.918098:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432666.918108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432666.918117:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913649664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432666.918124:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913649792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432666.918135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432666.918138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432666.918140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432666.918144:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5036 00000100:00080000:3.0:1789432666.918148:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432666.918152:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432666.918158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432666.918169:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913649920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432666.918179:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432666.918185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1789432666.918186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432666.918188:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432666.918195:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5036 00000100:00080000:3.0:1789432666.918209:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432666.918214:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432666.918221:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432666.918233:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913650048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432666.918246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432666/real 1789432666] req@ffff9811d1bc5880 x1876350913649664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432666.918267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913649664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432666.918278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432666.918282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432666.918285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913649664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432666.918308:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432666.918331:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432666.918342:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432666/real 1789432666] req@ffff9811d1bc5f80 x1876350913649792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432666.918353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913649792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432666.918367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432666.918370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432666.918372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432666.918378:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432666.918397:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432666.918424:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432666/real 1789432666] req@ffff9811d1bc4000 x1876350913649920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432666.918435:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913649920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432666.918442:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432666.918446:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432666.918449:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913649920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432666.918454:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650048:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432666.918473:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432666.918483:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432666/real 1789432666] req@ffff9811d1bc6300 x1876350913650048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432666.918492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913650048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432666.918499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432666.918502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432666.918505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650048:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432667.813989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432668.838013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432669.862076:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432670.886147:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432671.910076:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432672.037981:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432672.037996:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432672.038008:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432672.038021:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5041 00000100:00080000:2.0:1789432672.038033:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432672.038047:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432672.038095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913650176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432672.038124:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432672.038129:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432672.038131:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432672.038136:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5041 00000100:00080000:2.0:1789432672.038140:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432672.038146:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432672.038158:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432672.038177:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913650176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432672.038186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913650304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432672.038198:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432672.038202:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432672.038204:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432672.038209:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5041 00000100:00080000:2.0:1789432672.038213:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432672.038218:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432672.038223:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432672.038234:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913650432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432672.038243:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432672.038247:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432672.038249:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432672.038253:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5041 00000100:00080000:2.0:1789432672.038257:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432672.038261:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432672.038266:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789432672.038269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432672.038284:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913650560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432672.038337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432672/real 1789432672] req@ffff9811d2103b80 x1876350913650176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432672.038356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913650176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432672.038368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432672.038372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432672.038381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913650176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432672.038409:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432672.038432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432672.038443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432672/real 1789432672] req@ffff9811d2101c00 x1876350913650304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432672.038455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913650304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432672.038464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432672.038467:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432672.038470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432672.038478:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432672.038495:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432672.038505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432672/real 1789432672] req@ffff9811d2102a00 x1876350913650432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432672.038516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913650432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432672.038524:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432672.038527:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432672.038530:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432672.038536:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650560:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432672.038552:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432672.038560:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432672/real 1789432672] req@ffff9811d2100e00 x1876350913650560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432672.038571:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913650560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432672.038580:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432672.038583:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432672.038586:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432672.934095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432673.957989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432674.981994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432676.006028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432677.030033:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432677.158432:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432677.158442:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432677.158447:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432677.158457:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5046 00000100:00080000:1.0:1789432677.158466:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432677.158476:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432677.158514:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913650688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432677.158543:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432677.158547:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432677.158549:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432677.158553:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5046 00000100:00080000:1.0:1789432677.158556:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432677.158561:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432677.158571:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432677.158586:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913650816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432677.158589:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913650688:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432677.158604:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432677.158608:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432677.158610:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432677.158614:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5046 00000100:00080000:1.0:1789432677.158618:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432677.158623:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432677.158627:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432677.158638:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913650944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432677.158648:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789432677.158654:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432677.158654:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432677.158657:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432677.158663:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5046 00000100:00080000:1.0:1789432677.158667:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432677.158673:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432677.158678:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432677.158689:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913651072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432677.158692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432677/real 1789432677] req@ffff9811d0bded80 x1876350913650688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432677.158722:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913650688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432677.158732:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432677.158736:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432677.158739:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913650688:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432677.158758:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432677.158775:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432677.158785:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432677/real 1789432677] req@ffff9811d0bdc000 x1876350913650816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432677.158797:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913650816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432677.158804:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432677.158807:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432677.158810:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432677.158815:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432677.158828:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432677.158866:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432677/real 1789432677] req@ffff9811d0bdc700 x1876350913650944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432677.158884:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913650944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432677.158892:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432677.158895:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432677.158898:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913650944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432677.158903:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651072:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432677.158919:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432677.158927:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432677/real 1789432677] req@ffff9811d0bdd880 x1876350913651072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432677.158938:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913651072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432677.158945:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432677.158948:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432677.158951:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651072:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432678.054020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432679.078000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432680.102036:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432681.125968:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432682.150006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432682.277933:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432682.277943:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432682.277947:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432682.277957:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5051 00000100:00080000:1.0:1789432682.277966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432682.277978:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432682.278012:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913651200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432682.278035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432682.278039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432682.278041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432682.278046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5051 00000100:00080000:1.0:1789432682.278050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432682.278056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432682.278063:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913651200:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789432682.278066:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432682.278082:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913651328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432682.278107:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432682.278110:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432682.278112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432682.278117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5051 00000100:00080000:1.0:1789432682.278121:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432682.278129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789432682.278134:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789432682.278139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432682.278154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913651456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432682.278164:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432682.278168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432682.278169:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789432682.278175:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432682/real 1789432682] req@ffff9811d0bdf800 x1876350913651200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432682.278175:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5051 00000100:00080000:1.0:1789432682.278181:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432682.278191:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432682.278201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432682.278210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913651200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432682.278217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913651584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432682.278224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432682.278230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432682.278234:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913651200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432682.278253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432682.278270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432682.278280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432682/real 1789432682] req@ffff9811d0bde300 x1876350913651328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432682.278291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913651328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432682.278299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432682.278302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432682.278305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432682.278309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432682.278321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432682.278329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432682/real 1789432682] req@ffff9811d0bdf100 x1876350913651456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432682.278339:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913651456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432682.278353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432682.278355:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432682.278358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432682.278363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651584:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432682.278376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432682.278384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432682/real 1789432682] req@ffff9811d0bdd880 x1876350913651584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432682.278393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913651584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432682.278401:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432682.278404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432682.278415:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651584:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432683.174062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432684.198045:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432685.222104:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432686.246014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432687.270102:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432687.398048:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432687.398072:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432687.398077:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432687.398090:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5056 00000100:00080000:2.0:1789432687.398101:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432687.398115:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432687.398162:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913651712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432687.398191:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432687.398195:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432687.398197:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432687.398201:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5056 00000100:00080000:2.0:1789432687.398204:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432687.398214:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432687.398228:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432687.398242:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913651840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432687.398255:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432687.398272:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789432687.398274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913651712:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432687.398275:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432687.398288:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5056 00000100:00080000:2.0:1789432687.398293:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432687.398297:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432687.398302:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432687.398317:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913651968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432687.398327:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432687.398331:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432687.398334:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432687.398350:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5056 00000800:00000100:0.0:1789432687.398355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432687.398359:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432687.398370:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432687.398377:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432687.398397:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913652096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432687.398425:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432687/real 1789432687] req@ffff9811d2101880 x1876350913651712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432687.398444:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913651712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432687.398460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432687.398464:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432687.398468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913651712:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432687.398493:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432687.398519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432687.398532:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432687/real 1789432687] req@ffff9811d2101180 x1876350913651840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432687.398545:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913651840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432687.398554:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432687.398557:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432687.398561:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432687.398566:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432687.398585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432687.398596:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432687/real 1789432687] req@ffff9811d2101f80 x1876350913651968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432687.398607:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913651968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432687.398614:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432687.398617:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432687.398621:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913651968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432687.398628:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652096:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432687.398645:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432687.398655:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432687/real 1789432687] req@ffff9811d2100e00 x1876350913652096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432687.398671:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913652096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432687.398678:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432687.398681:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432687.398685:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652096:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432688.294086:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432689.318085:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432690.342017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432691.365985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432692.391067:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432692.517976:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432692.517986:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432692.517991:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432692.518001:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5062 00000100:00080000:0.0:1789432692.518010:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432692.518032:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432692.518065:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913652224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432692.518099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432692.518103:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432692.518105:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432692.518110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5062 00000100:00080000:0.0:1789432692.518113:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432692.518118:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432692.518132:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432692.518154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913652352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432692.518164:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432692.518168:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432692.518170:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432692.518174:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5062 00000100:00080000:0.0:1789432692.518177:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432692.518182:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432692.518187:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432692.518190:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913652224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432692.518200:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913652480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432692.518212:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432692.518215:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432692.518217:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432692.518220:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5062 00000100:00080000:0.0:1789432692.518236:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432692.518241:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432692.518246:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432692.518249:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432692.518262:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913652608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432692.518288:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432692/real 1789432692] req@ffff9811c8cd7480 x1876350913652224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432692.518323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913652224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432692.518333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432692.518338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432692.518341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913652224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432692.518348:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432692.518367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432692.518378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432692/real 1789432692] req@ffff9811c8cd7800 x1876350913652352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432692.518388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913652352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432692.518396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432692.518399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432692.518402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432692.518421:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432692.518437:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432692.518444:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432692/real 1789432692] req@ffff9811c8cd5180 x1876350913652480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432692.518451:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913652480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432692.518459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432692.518461:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432692.518463:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432692.518466:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652608:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432692.518476:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432692.518481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432692/real 1789432692] req@ffff9811c8cd4000 x1876350913652608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432692.518487:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913652608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432692.518492:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432692.518494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432692.518495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652608:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432693.413943:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432694.437981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432695.462008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432696.485994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432697.510014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432697.637948:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432697.637959:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432697.637964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432697.637973:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5067 00000100:00080000:3.0:1789432697.637991:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432697.638001:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432697.638040:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913652736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432697.638066:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432697.638068:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432697.638070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432697.638073:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5067 00000100:00080000:3.0:1789432697.638075:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432697.638079:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432697.638086:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432697.638099:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913652864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432697.638110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432697.638115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913652736:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432697.638115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432697.638118:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432697.638127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5067 00000100:00080000:3.0:1789432697.638131:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432697.638135:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432697.638141:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432697.638154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913652992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432697.638170:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432697.638173:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432697.638175:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432697.638181:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5067 00000800:00000100:1.0:1789432697.638182:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432697.638186:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432697.638191:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432697.638196:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432697.638205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913653120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432697.638226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432697/real 1789432697] req@ffff9811d1bc7480 x1876350913652736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432697.638244:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913652736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432697.638254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432697.638258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432697.638262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913652736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432697.638283:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432697.638306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432697.638318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432697/real 1789432697] req@ffff9811d1bc5c00 x1876350913652864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432697.638330:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913652864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432697.638337:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432697.638340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432697.638344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432697.638349:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432697.638366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432697.638376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432697/real 1789432697] req@ffff9811d1bc7800 x1876350913652992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432697.638385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913652992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432697.638391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432697.638394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432697.638397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913652992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432697.638402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653120:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432697.638433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432697.638445:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432697/real 1789432697] req@ffff9811d1bc5180 x1876350913653120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432697.638457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913653120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432697.638464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432697.638467:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432697.638470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653120:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432698.533963:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432699.558140:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432700.581994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432701.605993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432702.629979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432702.757927:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432702.757938:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432702.757953:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432702.757963:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5072 00000100:00080000:3.0:1789432702.757971:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432702.757980:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432702.758007:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913653248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432702.758028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432702.758030:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432702.758031:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432702.758034:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5072 00000100:00080000:3.0:1789432702.758036:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432702.758039:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432702.758045:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432702.758055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913653376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432702.758063:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432702.758065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432702.758066:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432702.758071:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5072 00000100:00100000:1.0:1789432702.758072:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913653248:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432702.758073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432702.758078:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432702.758083:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432702.758090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913653504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432702.758096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432702.758098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432702.758099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432702.758101:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5072 00000100:00080000:3.0:1789432702.758103:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432702.758105:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432702.758108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432702.758115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913653632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432702.758145:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432702.758183:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432702/real 1789432702] req@ffff9811d1bc6a00 x1876350913653248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432702.758201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913653248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432702.758208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432702.758210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432702.758213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913653248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432702.758228:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432702.758242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432702.758250:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432702/real 1789432702] req@ffff9811d1bc5180 x1876350913653376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432702.758257:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913653376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432702.758265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432702.758267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432702.758269:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432702.758272:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432702.758281:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432702.758286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432702/real 1789432702] req@ffff9811d1bc7800 x1876350913653504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432702.758292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913653504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432702.758297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432702.758298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432702.758300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432702.758303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653632:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432702.758313:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432702.758317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432702/real 1789432702] req@ffff9811d1bc5c00 x1876350913653632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432702.758324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913653632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432702.758328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432702.758330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432702.758332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653632:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432703.654006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432704.677991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432705.702033:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432706.725992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432707.749990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432707.877877:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432707.877887:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432707.877891:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432707.877900:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5077 00000100:00080000:3.0:1789432707.877918:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432707.877929:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432707.877969:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913653760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432707.877991:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432707.877994:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432707.877995:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432707.877998:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5077 00000100:00080000:3.0:1789432707.878001:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432707.878004:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432707.878011:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432707.878022:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913653888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432707.878030:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789432707.878031:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913653760:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432707.878032:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432707.878036:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432707.878042:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5077 00000100:00080000:3.0:1789432707.878044:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432707.878046:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432707.878049:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432707.878056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913654016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432707.878067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432707.878070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432707.878072:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432707.878076:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5077 00000100:00080000:3.0:1789432707.878080:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432707.878087:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432707.878091:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432707.878097:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432707.878107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913654144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432707.878134:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432707/real 1789432707] req@ffff9811d1bc7480 x1876350913653760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432707.878147:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913653760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432707.878154:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432707.878156:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432707.878159:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913653760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432707.878174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432707.878189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432707.878196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432707/real 1789432707] req@ffff9811d1bc6300 x1876350913653888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432707.878203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913653888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432707.878208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432707.878210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432707.878212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913653888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432707.878215:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432707.878223:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432707.878229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432707/real 1789432707] req@ffff9811d1bc4000 x1876350913654016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432707.878235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913654016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432707.878239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432707.878241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432707.878242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432707.878245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654144:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432707.878254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432707.878259:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432707/real 1789432707] req@ffff9811d1bc5f80 x1876350913654144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432707.878265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913654144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432707.878270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432707.878271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432707.878273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654144:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432708.774005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432709.797978:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432710.822018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432711.845981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432712.870000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432712.997983:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432712.997992:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432712.997996:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432712.998004:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5082 00000100:00080000:1.0:1789432712.998013:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432712.998022:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432712.998054:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913654272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432712.998080:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432712.998084:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432712.998086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432712.998091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5082 00000100:00080000:1.0:1789432712.998095:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432712.998100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432712.998106:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913654272:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789432712.998111:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432712.998126:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913654400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432712.998137:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432712.998141:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432712.998143:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432712.998146:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5082 00000100:00080000:1.0:1789432712.998150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789432712.998152:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432712.998160:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432712.998166:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432712.998178:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913654528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432712.998184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432712/real 1789432712] req@ffff9811d0bdc700 x1876350913654272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432712.998195:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432712.998199:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1789432712.998202:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913654272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432712.998203:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432712.998213:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5082 00000100:00080000:0.0:1789432712.998215:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432712.998217:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432712.998220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432712.998224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913654272:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432712.998228:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432712.998235:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432712.998243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432712.998249:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913654656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432712.998259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432712.998271:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432712/real 1789432712] req@ffff9811d0bdc000 x1876350913654400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432712.998279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913654400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432712.998284:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432712.998287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432712.998290:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432712.998293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432712.998302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432712.998307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432712/real 1789432712] req@ffff9811d0bded80 x1876350913654528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432712.998322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913654528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432712.998333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432712.998336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432712.998339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432712.998351:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654656:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432712.998371:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432712.998381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432712/real 1789432712] req@ffff9811d0bdf480 x1876350913654656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432712.998392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913654656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432712.998400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432712.998403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432712.998406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654656:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432713.893994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432714.918028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432715.942016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432716.966038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432717.990010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432718.118017:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432718.118027:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432718.118031:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432718.118040:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5087 00000100:00080000:3.0:1789432718.118049:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432718.118060:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432718.118095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913654784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432718.118125:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432718.118128:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432718.118130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432718.118134:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5087 00000100:00080000:3.0:1789432718.118139:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432718.118144:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432718.118157:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913654784:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789432718.118164:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432718.118176:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913654912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432718.118184:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432718.118186:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432718.118187:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432718.118190:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5087 00000100:00080000:3.0:1789432718.118192:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432718.118196:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432718.118199:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432718.118205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913655040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432718.118211:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432718.118215:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1789432718.118217:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432718.118217:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432718.118222:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5087 00000100:00080000:3.0:1789432718.118223:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432718.118226:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432718.118228:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432718.118235:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913655168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432718.118272:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432718/real 1789432718] req@ffff9811d1bc5880 x1876350913654784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432718.118291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913654784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432718.118300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432718.118304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432718.118307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913654784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432718.118391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432718.118411:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432718.118422:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432718/real 1789432718] req@ffff9811d1bc7100 x1876350913654912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432718.118432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913654912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432718.118444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432718.118448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432718.118450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913654912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432718.118455:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432718.118470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432718.118478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432718/real 1789432718] req@ffff9811d1bc6680 x1876350913655040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432718.118537:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913655040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432718.118545:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432718.118548:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432718.118550:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432718.118556:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655168:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432718.118575:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432718.118626:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432718/real 1789432718] req@ffff9811d1bc5500 x1876350913655168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432718.118639:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913655168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432718.118648:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432718.118650:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432718.118654:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655168:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432719.014003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432720.037982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432721.062194:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432722.086060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432723.110077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432723.237947:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432723.237959:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432723.237966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432723.237979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5092 00000100:00080000:3.0:1789432723.237991:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432723.238005:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432723.238050:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913655296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432723.238142:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913655296:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432723.238146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432723.238152:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432723.238154:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432723.238160:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5092 00000100:00080000:3.0:1789432723.238165:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432723.238171:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432723.238183:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432723.238201:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913655424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432723.238212:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432723.238216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432723.238218:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789432723.238226:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432723.238232:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5092 00000100:00080000:3.0:1789432723.238240:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432723.238246:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432723.238251:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432723.238263:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913655552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432723.238279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432723/real 1789432723] req@ffff9811d1bc4700 x1876350913655296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432723.238282:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432723.238292:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432723.238296:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432723.238302:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5092 00000100:00080000:3.0:1789432723.238306:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789432723.238310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913655296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432723.238312:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432723.238327:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789432723.238329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432723.238336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432723.238345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913655296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432723.238346:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913655680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432723.238377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432723.238402:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432723.238416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432723/real 1789432723] req@ffff9811d1bc5500 x1876350913655424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432723.238428:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913655424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432723.238437:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432723.238441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432723.238444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432723.238453:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432723.238470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432723.238478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432723/real 1789432723] req@ffff9811d1bc6680 x1876350913655552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432723.238490:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913655552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432723.238497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432723.238500:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432723.238503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432723.238508:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655680:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432723.238524:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432723.238531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432723/real 1789432723] req@ffff9811d1bc7100 x1876350913655680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432723.238538:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913655680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432723.238545:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432723.238547:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432723.238550:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655680:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432724.134046:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432725.158040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432726.181996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432727.206236:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432728.229997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432728.357950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432728.357960:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432728.358074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432728.358084:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5097 00000100:00080000:3.0:1789432728.358090:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432728.358099:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432728.358133:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913655808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432728.358164:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432728.358168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432728.358170:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432728.358174:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5097 00000100:00080000:3.0:1789432728.358178:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432728.358184:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432728.358193:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432728.358205:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913655936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432728.358215:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432728.358218:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432728.358220:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432728.358224:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5097 00000100:00080000:3.0:1789432728.358227:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432728.358232:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432728.358238:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432728.358245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913655808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432728.358250:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913656064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432728.358263:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432728.358266:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432728.358268:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432728.358272:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5097 00000100:00080000:3.0:1789432728.358275:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432728.358280:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432728.358285:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432728.358295:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913656192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432728.358347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432728.358384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432728.358405:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432729.254009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432730.277989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432731.301981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432731.430079:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432731.430120:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432731.430134:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432731.430177:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432731.430187:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432731.430195:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789432731.430266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432728/real 1789432731] req@ffff9811d1bc4000 x1876350913656064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432731.430288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913656064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432731.430299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432731.430304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432731.430308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432731.430316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432728/real 1789432731] req@ffff9811d1bc6300 x1876350913656192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432731.430330:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913656192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432731.430338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432731.430341:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432731.430344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656192:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789432731.430351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432728/real 1789432731] req@ffff9811d1bc5880 x1876350913655808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432731.430364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913655808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432731.430371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432731.430374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432731.430378:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913655808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432731.430384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432728/real 1789432731] req@ffff9811d1bc5f80 x1876350913655936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432731.430395:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913655936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432731.430402:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432731.430405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432731.430408:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913655936:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789432732.326064:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432733.350000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432733.477938:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432733.477946:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432733.477948:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432733.477957:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5103 00000100:00080000:0.0:1789432733.477972:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432733.477983:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432733.478009:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913656320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432733.478033:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432733.478036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432733.478037:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432733.478040:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5103 00000100:00080000:0.0:1789432733.478042:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432733.478045:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432733.478053:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432733.478061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913656448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432733.478068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432733.478070:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432733.478071:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432733.478073:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5103 00000100:00080000:0.0:1789432733.478076:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432733.478082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432733.478086:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432733.478093:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913656320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432733.478096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913656576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432733.478112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432733.478115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432733.478117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432733.478121:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5103 00000100:00080000:0.0:1789432733.478125:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432733.478130:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432733.478136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432733.478147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913656704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432733.478160:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432733.478200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432733/real 1789432733] req@ffff9811c8cd6300 x1876350913656320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432733.478218:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913656320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432733.478230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432733.478234:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432733.478237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913656320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432733.478244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432733.478266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432733.478280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432733/real 1789432733] req@ffff9811c8cd7b80 x1876350913656448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432733.478292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913656448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432733.478300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432733.478303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432733.478306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432733.478333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432733.478354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432733.478372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432733/real 1789432733] req@ffff9811c8cd6a00 x1876350913656576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432733.478381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913656576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432733.478388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432733.478392:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432733.478394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432733.478400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656704:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432733.478418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432733.478427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432733/real 1789432733] req@ffff9811c8cd4a80 x1876350913656704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432733.478438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913656704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432733.478445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432733.478449:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432733.478451:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656704:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432734.373981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432735.398260:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432736.422039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432737.445966:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432738.470898:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432738.598004:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432738.598013:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432738.598018:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432738.598026:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5108 00000100:00080000:3.0:1789432738.598035:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432738.598045:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432738.598078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913656832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432738.598109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432738.598112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432738.598114:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432738.598118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5108 00000100:00080000:3.0:1789432738.598122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432738.598126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432738.598135:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432738.598154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913656960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432738.598162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913656832:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432738.598168:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432738.598173:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432738.598174:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432738.598178:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5108 00000100:00080000:3.0:1789432738.598182:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432738.598186:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432738.598191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432738.598202:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913657088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432738.598217:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432738.598220:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1789432738.598225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432738.598225:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432738.598230:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5108 00000100:00080000:3.0:1789432738.598236:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432738.598241:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432738.598252:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432738.598264:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913657216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432738.598279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432738/real 1789432738] req@ffff9811d1bc7480 x1876350913656832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432738.598299:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913656832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432738.598309:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432738.598312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432738.598316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913656832:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432738.598343:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432738.598365:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432738.598377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432738/real 1789432738] req@ffff9811d1bc5c00 x1876350913656960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432738.598388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913656960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432738.598396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432738.598399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432738.598402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913656960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432738.598408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432738.598441:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432738.598452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432738/real 1789432738] req@ffff9811d1bc7800 x1876350913657088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432738.598464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913657088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432738.598472:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432738.598476:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432738.598478:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432738.598484:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657216:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432738.598503:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432738.598513:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432738/real 1789432738] req@ffff9811d1bc5180 x1876350913657216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432738.598524:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913657216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432738.598533:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432738.598536:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432738.598539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432739.494013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432740.518077:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432741.541968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432742.565985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432743.590002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432743.717955:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432743.717964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432743.717968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432743.717977:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5113 00000100:00080000:0.0:1789432743.717987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432743.717999:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432743.718031:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913657344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432743.718053:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432743.718055:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432743.718056:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432743.718059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5113 00000100:00080000:0.0:1789432743.718061:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432743.718065:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432743.718071:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432743.718081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913657472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432743.718088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432743.718091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432743.718093:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432743.718093:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913657344:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432743.718096:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5113 00000100:00080000:0.0:1789432743.718101:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432743.718104:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432743.718106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432743.718115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913657600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432743.718121:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432743.718123:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432743.718124:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432743.718127:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5113 00000100:00080000:0.0:1789432743.718138:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432743.718144:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432743.718149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432743.718150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432743.718165:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913657728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432743.718188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432743/real 1789432743] req@ffff9811c8cd5f80 x1876350913657344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432743.718201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913657344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432743.718209:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432743.718211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432743.718214:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913657344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432743.718231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432743.718245:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432743.718252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432743/real 1789432743] req@ffff9811c8cd6d80 x1876350913657472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432743.718259:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913657472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432743.718264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432743.718266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432743.718268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432743.718271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432743.718279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432743.718284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432743/real 1789432743] req@ffff9811c8cd5500 x1876350913657600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432743.718290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913657600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432743.718298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432743.718300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432743.718302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432743.718305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657728:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432743.718324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432743.718336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432743/real 1789432743] req@ffff9811c8cd4a80 x1876350913657728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432743.718346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913657728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432743.718353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432743.718356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432743.718359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657728:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432744.613988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432745.638003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432746.662332:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432747.686004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432748.710118:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432748.837936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432748.837945:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432748.837950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432748.837961:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5118 00000100:00080000:2.0:1789432748.837969:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432748.837978:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432748.838011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913657856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432748.838028:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432748.838030:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432748.838032:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432748.838034:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5118 00000100:00080000:2.0:1789432748.838036:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432748.838039:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432748.838045:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432748.838053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913657984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432748.838060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432748.838062:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432748.838063:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432748.838067:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5118 00000100:00100000:1.0:1789432748.838068:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913657856:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432748.838069:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432748.838075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432748.838078:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432748.838085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913658112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432748.838096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432748.838098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432748.838099:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432748.838101:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5118 00000100:00080000:2.0:1789432748.838103:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432748.838106:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432748.838108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432748.838115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913658240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432748.838128:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432748.838163:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432748/real 1789432748] req@ffff9811d2102a00 x1876350913657856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432748.838180:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913657856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432748.838191:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432748.838195:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432748.838223:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913657856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432748.838243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432748.838261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432748.838270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432748/real 1789432748] req@ffff9811d2101c00 x1876350913657984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432748.838279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913657984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432748.838285:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432748.838287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432748.838289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913657984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432748.838293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432748.838302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432748.838308:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432748/real 1789432748] req@ffff9811d2103b80 x1876350913658112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432748.838314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913658112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432748.838319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432748.838321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432748.838322:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432748.838325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658240:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432748.838334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432748.838339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432748/real 1789432748] req@ffff9811d2101500 x1876350913658240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432748.838346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913658240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432748.838351:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432748.838352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432748.838354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658240:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432749.734004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432750.758012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432751.781996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432752.805991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432753.830051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432753.957939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432753.957949:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432753.957953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432753.957962:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5123 00000100:00080000:2.0:1789432753.957970:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432753.957981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432753.958019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913658368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432753.958043:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432753.958045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432753.958047:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432753.958049:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5123 00000100:00080000:2.0:1789432753.958053:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432753.958056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432753.958063:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432753.958072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913658496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432753.958079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432753.958081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432753.958082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432753.958084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5123 00000100:00080000:2.0:1789432753.958086:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432753.958091:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432753.958092:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913658368:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789432753.958094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432753.958103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913658624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432753.958110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432753.958111:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432753.958112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432753.958115:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5123 00000100:00080000:2.0:1789432753.958124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432753.958127:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432753.958131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432753.958143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913658752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432753.958154:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432753.958207:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432753/real 1789432753] req@ffff9811d2102d80 x1876350913658368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432753.958226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913658368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432753.958238:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432753.958242:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432753.958245:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913658368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432753.958252:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432753.958271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432753.958282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432753/real 1789432753] req@ffff9811d2103800 x1876350913658496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432753.958294:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913658496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432753.958307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432753.958311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432753.958314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432753.958334:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432753.958353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432753.958363:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432753/real 1789432753] req@ffff9811d2100000 x1876350913658624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432753.958373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913658624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432753.958380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432753.958383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432753.958385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432753.958391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658752:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432753.958408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432753.958418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432753/real 1789432753] req@ffff9811d2101500 x1876350913658752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432753.958429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913658752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432753.958436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432753.958439:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432753.958443:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913658752:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432754.854060:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432755.878082:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432756.901989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432757.926004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432758.949992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432759.077950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432759.077962:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432759.077966:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432759.077976:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5128 00000100:00080000:3.0:1789432759.077985:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432759.077996:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432759.078026:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913658880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432759.078049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432759.078051:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432759.078053:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432759.078056:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5128 00000100:00080000:3.0:1789432759.078058:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432759.078061:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432759.078069:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432759.078079:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913659008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432759.078089:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432759.078092:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432759.078094:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432759.078098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913658880:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432759.078100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5128 00000100:00080000:3.0:1789432759.078105:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432759.078111:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432759.078115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432759.078127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913659136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432759.078134:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432759.078137:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432759.078139:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432759.078143:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5128 00000100:00080000:3.0:1789432759.078146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432759.078153:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432759.078158:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432759.078164:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789432759.078171:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913659264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432759.078206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432759/real 1789432759] req@ffff9811d1bc6a00 x1876350913658880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432759.078224:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913658880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432759.078234:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432759.078238:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432759.078242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913658880:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432759.078262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432759.078284:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432759.078297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432759/real 1789432759] req@ffff9811d1bc5180 x1876350913659008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432759.078308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913659008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432759.078316:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432759.078320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432759.078323:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432759.078328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432759.078347:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432759.078359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432759/real 1789432759] req@ffff9811d1bc7800 x1876350913659136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432759.078370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913659136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432759.078376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432759.078380:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432759.078383:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432759.078388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659264:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432759.078407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432759.078418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432759/real 1789432759] req@ffff9811d1bc5c00 x1876350913659264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432759.078429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913659264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432759.078436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432759.078439:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432759.078442:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659264:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432759.973996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432760.998673:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432762.022367:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432763.046110:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432764.070096:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432764.197936:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432764.197943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432764.197947:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432764.197956:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5133 00000100:00080000:3.0:1789432764.197964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432764.197973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432764.198007:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913659392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432764.198032:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432764.198034:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432764.198036:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432764.198038:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5133 00000100:00080000:3.0:1789432764.198041:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432764.198044:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432764.198051:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432764.198065:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913659520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432764.198076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432764.198079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432764.198085:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789432764.198087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913659392:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432764.198094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5133 00000100:00080000:3.0:1789432764.198104:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432764.198109:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432764.198115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432764.198131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913659648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432764.198147:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432764.198150:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432764.198152:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432764.198157:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5133 00000100:00080000:3.0:1789432764.198161:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789432764.198170:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432764.198171:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432764.198179:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432764.198207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913659776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432764.198214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432764/real 1789432764] req@ffff9811d1bc7480 x1876350913659392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432764.198244:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913659392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432764.198258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432764.198262:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432764.198266:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913659392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432764.198304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432764.198324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432764.198333:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432764/real 1789432764] req@ffff9811d1bc5f80 x1876350913659520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432764.198340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913659520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432764.198346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432764.198348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432764.198350:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432764.198354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432764.198364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432764.198369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432764/real 1789432764] req@ffff9811d1bc5880 x1876350913659648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432764.198375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913659648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432764.198380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432764.198381:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432764.198383:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432764.198387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659776:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432764.198398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432764.198403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432764/real 1789432764] req@ffff9811d1bc6300 x1876350913659776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432764.198410:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913659776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432764.198415:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432764.198416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432764.198418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913659776:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432765.093991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432766.117984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432767.141987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432768.166004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432769.189993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432769.317962:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432769.317972:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432769.317976:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432769.317986:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5138 00000100:00080000:3.0:1789432769.317996:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432769.318007:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432769.318047:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913659904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432769.318076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432769.318079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432769.318080:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432769.318083:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5138 00000100:00080000:3.0:1789432769.318086:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432769.318089:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432769.318100:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432769.318112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913660032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432769.318122:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432769.318126:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432769.318128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432769.318132:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5138 00000100:00080000:3.0:1789432769.318136:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789432769.318139:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913659904:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432769.318143:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432769.318149:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432769.318158:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913660160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432769.318166:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432769.318169:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432769.318171:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432769.318173:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5138 00000100:00080000:3.0:1789432769.318176:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432769.318179:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432769.318182:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432769.318196:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913660288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432769.318207:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432769.318257:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432769/real 1789432769] req@ffff9811d1bc4000 x1876350913659904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432769.318273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913659904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432769.318281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432769.318284:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432769.318287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913659904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432769.318292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432769.318308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432769.318316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432769/real 1789432769] req@ffff9811d1bc7100 x1876350913660032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432769.318324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913660032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432769.318329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432769.318331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432769.318333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432769.318352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432769.318361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432769.318366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432769/real 1789432769] req@ffff9811d1bc6680 x1876350913660160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432769.318373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913660160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432769.318377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432769.318379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432769.318380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432769.318384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660288:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432769.318394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432769.318398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432769/real 1789432769] req@ffff9811d1bc5500 x1876350913660288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432769.318409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913660288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432769.318423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432769.318426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432769.318429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660288:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432770.214057:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432771.238076:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432772.261972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432773.286078:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432774.310026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432774.437895:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432774.437904:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432774.437909:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432774.437918:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5144 00000100:00080000:3.0:1789432774.437926:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432774.437936:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432774.437965:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913660416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432774.437987:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432774.437989:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432774.437990:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432774.437993:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5144 00000100:00080000:3.0:1789432774.437995:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432774.437999:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432774.438006:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432774.438015:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913660544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432774.438021:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432774.438023:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432774.438024:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432774.438026:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5144 00000100:00080000:3.0:1789432774.438028:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432774.438034:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432774.438036:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913660416:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789432774.438037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432774.438050:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913660672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432774.438067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432774.438070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432774.438072:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432774.438076:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5144 00000100:00080000:3.0:1789432774.438080:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432774.438085:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432774.438089:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789432774.438093:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432774.438107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913660800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432774.438130:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432774/real 1789432774] req@ffff9811d1bc4700 x1876350913660416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432774.438145:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913660416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432774.438152:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432774.438155:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432774.438157:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913660416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432774.438162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432774.438177:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432774.438184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432774/real 1789432774] req@ffff9811d1bc5500 x1876350913660544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432774.438193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913660544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432774.438201:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432774.438205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432774.438209:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432774.438231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432774.438250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432774.438261:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432774/real 1789432774] req@ffff9811d1bc6680 x1876350913660672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432774.438272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913660672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432774.438279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432774.438283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432774.438286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432774.438291:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660800:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432774.438310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432774.438323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432774/real 1789432774] req@ffff9811d1bc7100 x1876350913660800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432774.438332:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913660800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432774.438337:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432774.438339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432774.438341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913660800:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432775.333990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432776.357969:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432777.381964:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432778.405963:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432779.429997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432779.557914:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432779.557924:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432779.557932:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432779.557942:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5149 00000100:00080000:2.0:1789432779.557950:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432779.557961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432779.557995:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913660928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432779.558023:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432779.558026:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432779.558028:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432779.558032:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5149 00000100:00080000:2.0:1789432779.558036:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432779.558040:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432779.558049:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432779.558065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913661056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432779.558076:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913660928:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432779.558084:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432779.558091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432779.558094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432779.558099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5149 00000100:00080000:2.0:1789432779.558103:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432779.558107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432779.558112:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432779.558125:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913661184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432779.558136:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432779.558139:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432779.558141:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432779.558144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5149 00000100:00080000:2.0:1789432779.558148:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432779.558153:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432779.558159:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432779.558160:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432779.558174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913661312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432779.558201:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432779/real 1789432779] req@ffff9811d2103b80 x1876350913660928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432779.558218:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913660928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432779.558228:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432779.558232:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432779.558235:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913660928:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432779.558256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432779.558278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432779.558290:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432779/real 1789432779] req@ffff9811d2101c00 x1876350913661056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432779.558302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913661056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432779.558311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432779.558314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432779.558318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432779.558323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432779.558341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432779.558352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432779/real 1789432779] req@ffff9811d2102a00 x1876350913661184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432779.558363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913661184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432779.558376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432779.558397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432779.558401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432779.558407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661312:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432779.558428:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432779.558438:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432779/real 1789432779] req@ffff9811d2100e00 x1876350913661312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432779.558448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913661312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432779.558463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432779.558466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432779.558469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661312:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432780.453940:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432781.477985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432782.501971:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432783.526085:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432784.550036:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432784.677983:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432784.677994:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432784.677998:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432784.678007:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5154 00000100:00080000:1.0:1789432784.678015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432784.678023:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432784.678053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913661440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432784.678080:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432784.678085:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432784.678086:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432784.678091:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5154 00000100:00100000:0.0:1789432784.678125:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913661440:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432784.678142:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432784.678149:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432784.678158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432784.678173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913661568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432784.678187:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432784.678187:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432784.678192:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432784.678195:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432784.678200:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5154 00000100:00080000:1.0:1789432784.678205:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432784.678210:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432784.678216:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432784.678228:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913661696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432784.678234:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432784/real 1789432784] req@ffff9811d0bdfb80 x1876350913661440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432784.678244:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432784.678248:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432784.678251:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432784.678256:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5154 00000100:00100000:0.0:1789432784.678260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913661440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432784.678260:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432784.678270:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00080000:0.0:1789432784.678271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432784.678279:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000008:00100000:1.0:1789432784.678279:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432784.678285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913661440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432784.678295:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913661824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432784.678328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432784.678363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432784.678373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432784/real 1789432784] req@ffff9811d0bdea00 x1876350913661568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432784.678384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913661568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432784.678391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432784.678394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432784.678400:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432784.678405:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432784.678421:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432784.678431:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432784/real 1789432784] req@ffff9811d0bde680 x1876350913661696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432784.678442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913661696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432784.678448:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432784.678451:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432784.678453:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432784.678458:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661824:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432784.678475:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432784.678486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432784/real 1789432784] req@ffff9811d0bdd500 x1876350913661824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432784.678496:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913661824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432784.678502:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432784.678505:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432784.678520:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913661824:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432785.574011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432786.598072:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432787.622051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432788.646016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432789.669993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432789.797932:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432789.797942:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432789.797946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432789.797955:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5159 00000100:00080000:0.0:1789432789.797964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432789.797974:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432789.798007:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913661952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432789.798033:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432789.798036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432789.798038:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432789.798042:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5159 00000100:00080000:0.0:1789432789.798046:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432789.798052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432789.798068:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432789.798071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913661952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432789.798086:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913662080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432789.798094:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432789.798096:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432789.798098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432789.798102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5159 00000100:00080000:0.0:1789432789.798105:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432789.798119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432789.798124:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789432789.798130:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432789.798137:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913662208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432789.798147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432789.798151:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432789.798153:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432789.798156:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5159 00000100:00080000:0.0:1789432789.798160:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432789.798165:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432789.798170:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432789/real 1789432789] req@ffff9811c8cd6a00 x1876350913661952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789432789.798172:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432789.798188:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913662336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432789.798194:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913661952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432789.798209:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432789.798217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432789.798224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913661952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432789.798250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432789.798271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432789.798280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432789/real 1789432789] req@ffff9811c8cd7b80 x1876350913662080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432789.798292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913662080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432789.798300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432789.798303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432789.798306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432789.798326:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432789.798346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432789.798358:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432789/real 1789432789] req@ffff9811c8cd6300 x1876350913662208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432789.798368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913662208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432789.798375:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432789.798378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432789.798382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432789.798387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662336:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432789.798407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432789.798417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432789/real 1789432789] req@ffff9811c8cd4000 x1876350913662336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432789.798427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913662336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432789.798436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432789.798438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432789.798441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662336:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432790.693990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432791.718015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432792.742036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432793.766359:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432794.789989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432794.917927:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432794.917934:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432794.917937:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432794.917945:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5164 00000100:00080000:3.0:1789432794.917953:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432794.917961:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432794.917993:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913662464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432794.918017:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432794.918020:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432794.918025:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432794.918029:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5164 00000100:00080000:3.0:1789432794.918032:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432794.918036:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432794.918043:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913662464:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789432794.918044:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432794.918056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913662592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432794.918064:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432794.918066:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432794.918067:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432794.918069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5164 00000100:00080000:3.0:1789432794.918071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432794.918073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432794.918076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432794.918082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913662720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432794.918088:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432794.918090:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432794.918092:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432794.918094:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5164 00000100:00080000:3.0:1789432794.918096:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432794.918098:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432794.918101:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432794.918107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913662848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432794.918144:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432794.918178:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432794.918202:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662848:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432795.814473:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432796.838023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432797.861972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432797.990182:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432797.990223:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432797.990238:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432797.990279:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432797.990288:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432797.990298:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789432797.990316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432794/real 1789432797] req@ffff9811d1bc6300 x1876350913662592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432797.990340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913662592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432797.990350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432797.990354:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432797.990358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432797.990365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432794/real 1789432797] req@ffff9811d1bc5880 x1876350913662720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432797.990375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913662720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432797.990382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432797.990384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432797.990387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432797.990393:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432794/real 1789432797] req@ffff9811d1bc5f80 x1876350913662848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432797.990403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913662848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432797.990411:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432797.990414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432797.990417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913662848:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789432797.990423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432794/real 1789432797] req@ffff9811d1bc4000 x1876350913662464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432797.990432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913662464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432797.990439:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432797.990441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432797.990444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913662464:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432798.885999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432799.909957:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432800.037949:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432800.037959:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432800.037964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432800.037974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5169 00000100:00080000:3.0:1789432800.037983:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432800.037992:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432800.038024:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913662976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432800.038048:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432800.038052:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432800.038054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432800.038059:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5169 00000100:00080000:3.0:1789432800.038062:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432800.038067:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432800.038076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432800.038083:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913662976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432800.038094:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913663104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432800.038105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432800.038109:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432800.038111:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432800.038114:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5169 00000100:00080000:3.0:1789432800.038119:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432800.038123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432800.038128:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789432800.038151:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432800.038185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432800/real 1789432800] req@ffff9811d1bc7480 x1876350913662976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432800.038203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913662976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432800.038213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432800.038217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432800.038220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913662976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432800.038251:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913663232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432800.038253:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663104:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789432800.038276:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432800.038279:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1789432800.038288:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432800.038293:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432800.038299:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5169 00000100:00100000:0.0:1789432800.038304:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432800/real 1789432800] req@ffff9811d1bc5c00 x1876350913663104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789432800.038304:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432800.038313:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432800.038322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913663104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789432800.038326:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789432800.038335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432800.038339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432800.038346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432800.038347:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913663360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432800.038367:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432800.038387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432800.038398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432800/real 1789432800] req@ffff9811d1bc7800 x1876350913663232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432800.038408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913663232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432800.038416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432800.038419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432800.038422:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432800.038431:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663360:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432800.038449:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432800.038458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432800/real 1789432800] req@ffff9811d1bc5180 x1876350913663360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432800.038475:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913663360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432800.038482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432800.038485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432800.038488:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663360:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432800.933996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432801.958003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432802.982017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432804.006002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432805.029997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432805.157939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432805.157947:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432805.157950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432805.157958:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5174 00000100:00080000:0.0:1789432805.157964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432805.157973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432805.158000:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913663488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432805.158021:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432805.158023:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432805.158024:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432805.158027:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5174 00000100:00080000:0.0:1789432805.158029:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432805.158031:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432805.158039:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432805.158046:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913663616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432805.158054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432805.158057:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1789432805.158057:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913663488:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432805.158060:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432805.158066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5174 00000100:00080000:0.0:1789432805.158068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432805.158070:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432805.158073:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432805.158079:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913663744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432805.158085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432805.158088:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432805.158089:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432805.158091:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5174 00000100:00080000:0.0:1789432805.158093:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432805.158095:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432805.158098:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432805.158104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913663872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432805.158122:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432805.158155:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432805/real 1789432805] req@ffff9811c8cd5180 x1876350913663488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432805.158169:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913663488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432805.158177:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432805.158180:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432805.158183:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913663488:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432805.158214:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432805.158237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432805.158250:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432805/real 1789432805] req@ffff9811c8cd7800 x1876350913663616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432805.158262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913663616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432805.158276:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432805.158279:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432805.158283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432805.158289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432805.158303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432805.158310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432805/real 1789432805] req@ffff9811c8cd7480 x1876350913663744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432805.158316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913663744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432805.158321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432805.158323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432805.158325:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432805.158328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663872:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432805.158353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432805.158364:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432805/real 1789432805] req@ffff9811c8cd7100 x1876350913663872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432805.158375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913663872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432805.158383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432805.158386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432805.158389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913663872:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432806.053997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432807.077977:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432808.102050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432809.125980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432810.149982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432810.277961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432810.277970:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432810.277974:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432810.277984:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5179 00000100:00080000:1.0:1789432810.277993:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432810.278003:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432810.278037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913664000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432810.278062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432810.278065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432810.278067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432810.278072:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5179 00000100:00080000:1.0:1789432810.278075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432810.278080:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432810.278090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432810.278099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913664000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432810.278105:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913664128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432810.278116:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432810.278120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432810.278122:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432810.278125:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5179 00000100:00080000:1.0:1789432810.278129:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432810.278134:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432810.278139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432810.278155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913664256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432810.278171:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432810.278183:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432810.278187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432810.278189:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432810.278193:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5179 00000100:00080000:1.0:1789432810.278197:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432810.278202:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432810.278207:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432810.278217:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432810/real 1789432810] req@ffff9811d0bddc00 x1876350913664000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432810.278222:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913664384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432810.278244:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913664000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432810.278256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432810.278259:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432810.278262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913664000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432810.278277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432810.278293:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432810.278300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432810/real 1789432810] req@ffff9811d0bdc380 x1876350913664128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432810.278307:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913664128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432810.278312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432810.278315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432810.278318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432810.278323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432810.278341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432810.278352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432810/real 1789432810] req@ffff9811d0bdd180 x1876350913664256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432810.278367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913664256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432810.278374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432810.278377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432810.278380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432810.278386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664384:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432810.278404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432810.278415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432810/real 1789432810] req@ffff9811d0bdd500 x1876350913664384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432810.278425:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913664384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432810.278432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432810.278436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432810.278438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664384:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432811.173995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432812.198032:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432813.222032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432814.246015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432815.270000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432815.397992:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432815.398000:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432815.398004:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432815.398026:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5184 00000100:00080000:3.0:1789432815.398033:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432815.398043:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432815.398074:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913664512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432815.398097:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432815.398100:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432815.398102:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432815.398106:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5184 00000100:00080000:3.0:1789432815.398110:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432815.398115:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432815.398135:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432815.398149:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913664640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432815.398154:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913664512:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789432815.398168:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432815.398177:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432815.398179:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432815.398183:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5184 00000100:00080000:3.0:1789432815.398187:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432815.398192:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432815.398196:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432815.398208:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913664768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432815.398216:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432815.398223:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432815.398227:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432815.398229:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432815.398234:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5184 00000100:00080000:3.0:1789432815.398239:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432815.398243:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432815.398248:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432815.398257:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432815/real 1789432815] req@ffff9811d1bc6a00 x1876350913664512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432815.398268:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913664896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432815.398276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913664512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432815.398289:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432815.398292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432815.398296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913664512:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432815.398316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432815.398435:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432815.398448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432815/real 1789432815] req@ffff9811d1bc5180 x1876350913664640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432815.398460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913664640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432815.398468:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432815.398471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432815.398474:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432815.398480:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432815.398499:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432815.398509:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432815/real 1789432815] req@ffff9811d1bc7800 x1876350913664768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432815.398520:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913664768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432815.398527:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432815.398530:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432815.398532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432815.398538:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664896:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432815.398552:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432815.398559:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432815/real 1789432815] req@ffff9811d1bc5c00 x1876350913664896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432815.398566:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913664896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432815.398571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432815.398573:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432815.398575:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913664896:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432816.293999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432817.317982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432818.341960:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432819.365957:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432820.389975:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432820.518045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432820.518052:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432820.518056:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432820.518063:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5190 00000100:00080000:0.0:1789432820.518069:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432820.518077:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432820.518116:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913665024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432820.518146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432820.518150:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432820.518152:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432820.518156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5190 00000100:00080000:0.0:1789432820.518160:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432820.518165:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432820.518174:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432820.518183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913665152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432820.518190:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432820.518192:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432820.518193:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432820.518195:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5190 00000100:00080000:0.0:1789432820.518197:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432820.518200:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432820.518202:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432820.518207:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913665280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432820.518213:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432820.518215:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432820.518216:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432820.518218:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5190 00000100:00080000:0.0:1789432820.518219:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432820.518221:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432820.518223:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432820.518230:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913665408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432820.518298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913665024:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789432820.518353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432820.518397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432820/real 1789432820] req@ffff9811c8cd6680 x1876350913665024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432820.518414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913665024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432820.518424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432820.518427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432820.518431:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913665024:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432820.518440:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432820.518460:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432820.518472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432820/real 1789432820] req@ffff9811c8cd4700 x1876350913665152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432820.518485:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913665152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432820.518492:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432820.518495:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432820.518498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432820.518504:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432820.518532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432820.518543:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432820/real 1789432820] req@ffff9811c8cd5c00 x1876350913665280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432820.518554:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913665280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432820.518561:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432820.518564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432820.518568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432820.518575:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665408:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432820.518597:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432820.518606:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432820/real 1789432820] req@ffff9811c8cd7100 x1876350913665408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432820.518617:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913665408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432820.518625:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432820.518628:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432820.518631:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665408:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432821.414009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432822.437990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432823.462027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432824.486030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432825.510030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432825.637964:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432825.637972:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432825.637975:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432825.637983:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5195 00000100:00080000:1.0:1789432825.637990:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432825.637997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432825.638032:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913665536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432825.638063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432825.638065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432825.638066:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432825.638069:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5195 00000100:00080000:1.0:1789432825.638071:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432825.638075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432825.638082:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432825.638092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913665664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432825.638098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913665536:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432825.638111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432825.638113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432825.638115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432825.638118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5195 00000100:00080000:1.0:1789432825.638120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432825.638122:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432825.638125:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432825.638132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913665792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432825.638139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432825.638141:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432825.638142:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432825.638144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5195 00000100:00080000:1.0:1789432825.638147:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789432825.638150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432825.638153:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432825.638157:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432825.638165:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913665920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432825.638188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432825/real 1789432825] req@ffff9811d0bde680 x1876350913665536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432825.638201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913665536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432825.638208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432825.638211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432825.638215:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913665536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432825.638229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432825.638242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432825.638249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432825/real 1789432825] req@ffff9811d0bdea00 x1876350913665664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432825.638256:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913665664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432825.638261:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432825.638262:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432825.638265:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432825.638268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432825.638277:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432825.638281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432825/real 1789432825] req@ffff9811d0bdfb80 x1876350913665792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432825.638288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913665792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432825.638292:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432825.638293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432825.638295:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432825.638297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432825.638306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432825.638311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432825/real 1789432825] req@ffff9811d0bdf480 x1876350913665920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432825.638316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913665920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432825.638321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432825.638322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432825.638324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913665920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432826.533999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432827.558033:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432828.582068:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432829.606072:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432830.629995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432830.758050:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432830.758059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432830.758063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432830.758072:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5200 00000100:00080000:3.0:1789432830.758079:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432830.758090:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432830.758124:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913666048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432830.758148:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432830.758151:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432830.758152:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432830.758155:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5200 00000100:00080000:3.0:1789432830.758158:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432830.758164:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432830.758173:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432830.758180:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913666048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432830.758190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913666176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432830.758202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432830.758206:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432830.758207:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432830.758210:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5200 00000100:00080000:3.0:1789432830.758213:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432830.758216:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432830.758220:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432830.758231:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913666304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432830.758245:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432830.758251:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432830.758255:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432830.758257:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432830.758262:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5200 00000100:00080000:3.0:1789432830.758267:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432830.758271:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432830.758276:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432830.758282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432830/real 1789432830] req@ffff9811d1bc7480 x1876350913666048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789432830.758290:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913666432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432830.758313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913666048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432830.758321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432830.758324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432830.758326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913666048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432830.758342:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432830.758360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432830.758373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432830/real 1789432830] req@ffff9811d1bc4000 x1876350913666176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432830.758384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913666176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432830.758392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432830.758395:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432830.758398:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432830.758404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432830.758422:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432830.758432:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432830/real 1789432830] req@ffff9811d1bc5f80 x1876350913666304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432830.758442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913666304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432830.758454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432830.758457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432830.758460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432830.758466:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432830.758485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432830.758499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432830/real 1789432830] req@ffff9811d1bc5880 x1876350913666432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432830.758509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913666432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432830.758517:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432830.758520:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432830.758523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432831.654048:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432832.677982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432833.702011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432834.726015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432835.749990:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432835.877952:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432835.877961:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432835.877965:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432835.877974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5205 00000100:00080000:1.0:1789432835.877994:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432835.878005:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432835.878039:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913666560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432835.878118:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432835.878122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432835.878124:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432835.878128:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5205 00000100:00080000:1.0:1789432835.878133:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432835.878138:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432835.878147:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432835.878156:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913666560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432835.878163:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913666688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432835.878174:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432835.878178:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432835.878180:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432835.878183:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5205 00000100:00080000:1.0:1789432835.878187:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432835.878191:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432835.878195:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432835.878206:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913666816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432835.878216:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789432835.878219:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432835.878221:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432835.878224:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432835.878229:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5205 00000100:00080000:1.0:1789432835.878232:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432835.878238:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432835.878243:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432835.878254:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913666944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432835.878269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432835/real 1789432835] req@ffff9811d0bded80 x1876350913666560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432835.878290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913666560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432835.878299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432835.878303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432835.878307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913666560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432835.878327:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432835.878348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432835.878359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432835/real 1789432835] req@ffff9811d0bdc000 x1876350913666688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432835.878370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913666688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432835.878377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432835.878379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432835.878382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432835.878387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432835.878403:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432835.878417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432835/real 1789432835] req@ffff9811d0bdc700 x1876350913666816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432835.878426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913666816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432835.878433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432835.878435:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432835.878438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432835.878444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666944:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432835.878461:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432835.878471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432835/real 1789432835] req@ffff9811d0bdd880 x1876350913666944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432835.878481:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913666944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432835.878489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432835.878492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432835.878495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913666944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432836.774021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432837.798052:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432838.821963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432839.846212:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432840.870040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432840.997923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432840.997934:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432840.997948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432840.997962:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5210 00000100:00080000:1.0:1789432840.997976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432840.997995:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432840.998042:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913667072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432840.998073:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432840.998077:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432840.998079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432840.998083:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5210 00000100:00080000:1.0:1789432840.998087:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432840.998091:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432840.998102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432840.998115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913667200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432840.998126:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789432840.998128:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913667072:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432840.998131:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432840.998136:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432840.998141:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5210 00000100:00080000:1.0:1789432840.998144:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432840.998148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432840.998153:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432840.998165:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913667328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432840.998174:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432840.998177:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432840.998179:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432840.998183:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5210 00000100:00080000:1.0:1789432840.998195:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432840.998200:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432840.998205:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789432840.998212:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432840.998220:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913667456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432840.998260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432840/real 1789432840] req@ffff9811d0bdf100 x1876350913667072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432840.998281:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913667072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432840.998294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432840.998298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432840.998302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913667072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432840.998328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432840.998351:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432840.998360:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432840/real 1789432840] req@ffff9811d0bde300 x1876350913667200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432840.998371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913667200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432840.998380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432840.998383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432840.998386:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432840.998392:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432840.998407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432840.998413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432840/real 1789432840] req@ffff9811d0bdf800 x1876350913667328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432840.998421:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913667328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432840.998426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432840.998428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432840.998429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432840.998433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432840.998443:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432840.998448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432840/real 1789432840] req@ffff9811d0bdd880 x1876350913667456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432840.998454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913667456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432840.998463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432840.998465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432840.998467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432841.894032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432842.918066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432843.942077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432844.966528:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432845.990036:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432846.117995:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432846.118009:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432846.118015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432846.118028:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5215 00000100:00080000:3.0:1789432846.118039:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432846.118052:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432846.118115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913667584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432846.118153:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432846.118158:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432846.118160:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432846.118166:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5215 00000100:00080000:3.0:1789432846.118170:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432846.118176:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432846.118188:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432846.118203:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913667712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432846.118214:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432846.118218:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432846.118221:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432846.118225:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5215 00000100:00080000:3.0:1789432846.118230:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432846.118236:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432846.118244:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432846.118245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913667584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432846.118259:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913667840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432846.118272:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432846.118275:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432846.118277:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432846.118281:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5215 00000100:00080000:3.0:1789432846.118284:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432846.118288:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432846.118294:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432846.118305:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913667968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432846.118322:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432846.118413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432846/real 1789432846] req@ffff9811d1bc6300 x1876350913667584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432846.118433:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913667584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432846.118444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432846.118448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432846.118452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913667584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432846.118460:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432846.118485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432846.118497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432846/real 1789432846] req@ffff9811d1bc7100 x1876350913667712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432846.118508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913667712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432846.118516:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432846.118520:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432846.118523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432846.118544:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432846.118571:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432846.118581:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432846/real 1789432846] req@ffff9811d1bc6680 x1876350913667840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432846.118592:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913667840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432846.118599:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432846.118602:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432846.118605:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432846.118611:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667968:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432846.118628:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432846.118637:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432846/real 1789432846] req@ffff9811d1bc5500 x1876350913667968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432846.118648:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913667968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432846.118657:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432846.118660:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432846.118663:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913667968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432847.014455:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432848.038036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432849.062027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432850.085990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432851.110077:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432851.238003:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432851.238010:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432851.238015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432851.238022:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5220 00000100:00080000:3.0:1789432851.238029:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432851.238037:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432851.238064:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913668096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432851.238083:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432851.238085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432851.238086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432851.238089:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5220 00000100:00080000:3.0:1789432851.238091:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432851.238093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432851.238099:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432851.238108:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913668224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432851.238114:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432851.238116:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432851.238117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432851.238119:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5220 00000100:00080000:3.0:1789432851.238121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432851.238123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432851.238126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432851.238134:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913668096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432851.238134:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913668352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432851.238157:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432851.238160:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432851.238162:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432851.238166:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5220 00000100:00080000:3.0:1789432851.238169:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432851.238174:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432851.238178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432851.238190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913668480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432851.238194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432851.238235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432851/real 1789432851] req@ffff9811d1bc4700 x1876350913668096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432851.238252:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913668096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432851.238263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432851.238267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432851.238271:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913668096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432851.238278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432851.238299:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432851.238311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432851/real 1789432851] req@ffff9811d1bc5500 x1876350913668224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432851.238323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913668224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432851.238332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432851.238335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432851.238338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432851.238357:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432851.238376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432851.238387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432851/real 1789432851] req@ffff9811d1bc6680 x1876350913668352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432851.238406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913668352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432851.238413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432851.238416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432851.238419:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432851.238424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432851.238443:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432851.238453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432851/real 1789432851] req@ffff9811d1bc7100 x1876350913668480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432851.238464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913668480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432851.238472:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432851.238474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432851.238477:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432852.134000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432853.158037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432854.182003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432855.206009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432856.230038:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432856.357968:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432856.357978:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432856.357983:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432856.357993:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5225 00000100:00080000:3.0:1789432856.358003:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432856.358015:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432856.358055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913668608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432856.358098:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432856.358102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432856.358104:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432856.358109:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5225 00000100:00080000:3.0:1789432856.358113:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432856.358120:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432856.358133:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913668608:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789432856.358134:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432856.358153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913668736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432856.358165:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432856.358168:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432856.358171:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432856.358175:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5225 00000100:00080000:3.0:1789432856.358178:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432856.358183:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432856.358187:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432856.358195:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913668864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432856.358202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432856.358205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432856.358206:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432856.358209:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5225 00000800:00000100:0.0:1789432856.358214:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789432856.358214:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432856.358219:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432856.358223:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789432856.358230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913668992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432856.358260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432856/real 1789432856] req@ffff9811d1bc6300 x1876350913668608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432856.358275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913668608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432856.358283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432856.358287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432856.358289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913668608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432856.358309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432856.358328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432856.358339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432856/real 1789432856] req@ffff9811d1bc5880 x1876350913668736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432856.358351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913668736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432856.358359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432856.358362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432856.358365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432856.358371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432856.358390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432856.358401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432856/real 1789432856] req@ffff9811d1bc5f80 x1876350913668864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432856.358412:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913668864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432856.358420:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432856.358424:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432856.358426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432856.358435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432856.358450:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432856.358457:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432856/real 1789432856] req@ffff9811d1bc4000 x1876350913668992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432856.358464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913668992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432856.358470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432856.358472:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432856.358474:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913668992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432857.254010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432858.277968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432859.302075:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432860.325979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432861.350008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789432861.477901:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432861.477909:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789432861.477913:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432861.477921:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5231 00000100:00080000:3.0:1789432861.477928:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432861.477935:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789432861.477968:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913669120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432861.477995:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432861.477998:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789432861.478000:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432861.478004:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5231 00000100:00080000:3.0:1789432861.478008:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432861.478014:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432861.478023:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432861.478033:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913669120:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789432861.478037:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913669248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432861.478049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432861.478053:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789432861.478055:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432861.478058:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5231 00000100:00080000:3.0:1789432861.478062:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432861.478067:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432861.478072:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789432861.478084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913669376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789432861.478100:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789432861.478103:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789432861.478105:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789432861.478109:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5231 00000100:00080000:3.0:1789432861.478113:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789432861.478119:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789432861.478137:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432861.478140:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789432861.478153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913669504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432861.478174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432861.478210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432862.374014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432863.398011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432864.422000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432864.550082:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432864.550121:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432864.550136:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432864.550173:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432864.550182:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432864.550190:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789432864.551041:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432861/real 1789432864] req@ffff9811d1bc5180 x1876350913669504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432864.551072:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913669504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432864.551084:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432864.551089:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432864.551093:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669504:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789432864.551099:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432861/real 1789432864] req@ffff9811d1bc5c00 x1876350913669248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432864.551109:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913669248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432864.551115:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432864.551118:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432864.551120:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432864.551127:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432861/real 1789432864] req@ffff9811d1bc7800 x1876350913669376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432864.551138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913669376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432864.551144:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432864.551147:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432864.551150:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432864.551156:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432861/real 1789432864] req@ffff9811d1bc7480 x1876350913669120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432864.551167:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913669120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432864.551174:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432864.551178:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432864.551180:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913669120:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789432865.446002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432866.470032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432866.597959:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432866.597967:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432866.597971:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432866.597978:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5236 00000100:00080000:2.0:1789432866.597985:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432866.597994:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432866.598023:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913669632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432866.598053:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432866.598056:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432866.598057:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432866.598061:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5236 00000100:00080000:2.0:1789432866.598065:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432866.598077:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432866.598087:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432866.598099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913669760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432866.598109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432866.598113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432866.598114:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432866.598118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5236 00000100:00080000:2.0:1789432866.598122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432866.598127:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432866.598132:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432866.598136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913669632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432866.598145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913669888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432866.598157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432866.598166:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432866.598168:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432866.598172:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5236 00000100:00080000:2.0:1789432866.598177:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432866.598191:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789432866.598195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432866.598199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432866.598215:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913670016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432866.598241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432866/real 1789432866] req@ffff9811d2101f80 x1876350913669632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432866.598260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913669632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432866.598270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432866.598275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432866.598278:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913669632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432866.598285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432866.598312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432866.598324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432866/real 1789432866] req@ffff9811d2101180 x1876350913669760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432866.598335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913669760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432866.598343:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432866.598346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432866.598349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432866.598371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432866.598391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432866.598408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432866/real 1789432866] req@ffff9811d2101880 x1876350913669888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432866.598419:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913669888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432866.598426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432866.598429:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432866.598431:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913669888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432866.598441:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432866.598461:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432866.598472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432866/real 1789432866] req@ffff9811d2100e00 x1876350913670016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432866.598483:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913670016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432866.598490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432866.598493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432866.598496:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432867.494849:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432868.518130:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432869.542110:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432870.566031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432871.590013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432871.717987:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432871.717997:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432871.718002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432871.718011:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5241 00000100:00080000:0.0:1789432871.718028:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432871.718041:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432871.718075:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913670144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432871.718102:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432871.718106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432871.718107:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432871.718112:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5241 00000100:00080000:0.0:1789432871.718116:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432871.718122:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432871.718131:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432871.718144:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913670272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432871.718151:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913670144:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789432871.718155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432871.718158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432871.718160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432871.718162:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5241 00000100:00080000:0.0:1789432871.718165:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432871.718167:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432871.718171:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432871.718177:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913670400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432871.718188:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432871.718191:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432871.718192:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432871.718194:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5241 00000100:00080000:0.0:1789432871.718199:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432871.718201:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432871.718204:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432871.718210:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913670528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432871.718222:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432871.718258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432871/real 1789432871] req@ffff9811c8cd7480 x1876350913670144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432871.718280:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913670144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432871.718289:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432871.718294:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432871.718297:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913670144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432871.718317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432871.718339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432871.718351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432871/real 1789432871] req@ffff9811c8cd7800 x1876350913670272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432871.718362:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913670272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432871.718370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432871.718373:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432871.718376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432871.718381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432871.718398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432871.718409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432871/real 1789432871] req@ffff9811c8cd5180 x1876350913670400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432871.718429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913670400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432871.718440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432871.718443:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432871.718446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432871.718451:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670528:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432871.718471:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432871.718481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432871/real 1789432871] req@ffff9811c8cd4000 x1876350913670528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432871.718491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913670528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432871.718499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432871.718502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432871.718505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432872.614000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432873.638158:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432874.663261:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432875.686180:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432876.710177:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432876.837992:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432876.838002:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432876.838006:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432876.838016:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5246 00000100:00080000:1.0:1789432876.838023:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432876.838034:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432876.838070:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913670656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432876.838136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913670656:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789432876.838197:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432876.838258:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432876.838263:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432876.838265:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432876.838270:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5246 00000100:00080000:1.0:1789432876.838274:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432876.838280:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432876.838290:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432876.838306:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913670784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432876.838365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432876/real 1789432876] req@ffff9811d0bdc700 x1876350913670656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432876.838382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913670656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432876.838393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432876.838397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432876.838400:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913670656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432876.838433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432876.838456:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432876.838469:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432876/real 1789432876] req@ffff9811d0bdc000 x1876350913670784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432876.838481:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913670784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432876.838489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432876.838492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432876.838495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670784:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789432876.838540:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432876.838544:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432876.838546:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432876.838550:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5246 00000100:00080000:1.0:1789432876.838555:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432876.838560:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432876.838566:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432876.838578:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913670912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432876.838589:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432876.838592:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432876.838594:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432876.838598:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5246 00000100:00080000:1.0:1789432876.838601:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432876.838606:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432876.838611:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432876.838622:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913671040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432876.838802:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432876.838827:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432876.838860:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432876/real 1789432876] req@ffff9811d0bded80 x1876350913670912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432876.838872:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913670912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432876.838880:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432876.838883:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432876.838886:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913670912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432876.838893:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432876.838913:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432876.838925:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432876/real 1789432876] req@ffff9811d0bdf480 x1876350913671040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432876.838943:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913671040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432876.838951:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432876.838953:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432876.838956:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432877.734742:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432878.758107:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432879.782706:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432880.806000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432881.830056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432881.958494:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432881.958504:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432881.958507:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432881.958516:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5251 00000100:00080000:2.0:1789432881.958525:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432881.958536:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432881.958570:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913671168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432881.958970:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432881.958974:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432881.958976:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432881.959107:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5251 00000100:00080000:2.0:1789432881.959112:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432881.959119:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432881.959129:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432881.959143:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913671296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432881.959154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432881.959157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432881.959159:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432881.959162:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5251 00000100:00080000:2.0:1789432881.959165:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432881.959170:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432881.959174:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432881.959184:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913671424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432881.959205:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432881.959208:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432881.959210:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432881.959214:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5251 00000100:00080000:2.0:1789432881.959217:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432881.959221:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432881.959226:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432881.959237:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913671552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432881.959476:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913671168:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789432881.959534:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432881.959569:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432881/real 1789432881] req@ffff9811d2102a00 x1876350913671168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432881.959587:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913671168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432881.959597:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432881.959600:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432881.959604:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913671168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432881.959625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432881.959663:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432881.960451:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432881/real 1789432881] req@ffff9811d2101c00 x1876350913671296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432881.960474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913671296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432881.960483:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432881.960486:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432881.960490:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432881.960496:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432881.960521:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432881.960533:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432881/real 1789432881] req@ffff9811d2103b80 x1876350913671424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432881.960544:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913671424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432881.960551:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432881.960554:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432881.960557:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432881.960564:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432881.960583:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432881.960594:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432881/real 1789432881] req@ffff9811d2101500 x1876350913671552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432881.960605:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913671552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432881.960618:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432881.960621:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432881.960624:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432882.854217:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432883.880311:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432884.902582:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432885.926009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432886.950088:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432887.078117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432887.078128:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432887.078132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432887.078141:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5256 00000100:00080000:1.0:1789432887.078150:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432887.078161:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432887.078197:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913671680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432887.078223:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432887.078227:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432887.078229:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432887.078233:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5256 00000100:00080000:1.0:1789432887.078236:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432887.078239:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432887.078247:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432887.078257:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913671808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432887.078265:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432887.078267:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432887.078269:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432887.078273:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5256 00000100:00100000:0.0:1789432887.078274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913671680:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432887.078275:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432887.078281:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432887.078286:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432887.078293:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913671936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432887.078299:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432887.078301:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432887.078302:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432887.078304:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5256 00000100:00080000:1.0:1789432887.078306:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432887.078308:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432887.078311:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432887.078318:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913672064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432887.078340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432887.078376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432887/real 1789432887] req@ffff9811d0bdfb80 x1876350913671680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432887.078392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913671680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432887.078399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432887.078402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432887.078404:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913671680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432887.078430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432887.078446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432887.078453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432887/real 1789432887] req@ffff9811d0bdea00 x1876350913671808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432887.078460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913671808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432887.078469:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432887.078471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432887.078473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432887.078476:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432887.078485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432887.078490:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432887/real 1789432887] req@ffff9811d0bde680 x1876350913671936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432887.078497:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913671936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432887.078504:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432887.078507:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432887.078510:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913671936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432887.078515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672064:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432887.078533:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432887.078543:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432887/real 1789432887] req@ffff9811d0bdd500 x1876350913672064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432887.078554:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913672064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432887.078562:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432887.078565:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432887.078568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432887.974046:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432888.998803:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432890.022047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432891.046074:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432892.070041:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432892.198597:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432892.198609:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432892.198614:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432892.198625:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5261 00000100:00080000:2.0:1789432892.198635:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432892.198647:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432892.198690:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913672192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432892.198722:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432892.198725:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432892.198728:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432892.198731:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5261 00000100:00080000:2.0:1789432892.198734:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432892.198737:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432892.198746:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432892.198755:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913672320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432892.198761:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432892.198763:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432892.198764:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432892.198766:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5261 00000100:00080000:2.0:1789432892.198767:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432892.198769:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432892.198772:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432892.198777:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913672448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432892.198783:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432892.198785:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432892.198787:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432892.198789:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5261 00000100:00080000:2.0:1789432892.198791:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432892.198793:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432892.198795:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432892.198803:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913672576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432892.198913:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913672192:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789432892.198990:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432892.199040:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432892/real 1789432892] req@ffff9811d2100000 x1876350913672192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432892.199060:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913672192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432892.199072:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432892.199076:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432892.199080:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913672192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432892.199093:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432892.199117:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432892.199132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432892/real 1789432892] req@ffff9811d2103800 x1876350913672320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432892.199144:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913672320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432892.199152:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432892.199156:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432892.199159:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432892.199168:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432892.199188:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432892.199200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432892/real 1789432892] req@ffff9811d2102d80 x1876350913672448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432892.199210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913672448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432892.199217:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432892.199221:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432892.199237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432892.199245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432892.199266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432892.199277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432892/real 1789432892] req@ffff9811d2101500 x1876350913672576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432892.199287:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913672576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432892.199295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432892.199298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432892.199300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432893.094182:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432894.118066:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432895.142722:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432896.166041:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432897.190016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432897.320440:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432897.320455:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432897.320458:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432897.320468:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5266 00000100:00080000:0.0:1789432897.320477:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432897.320487:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432897.320522:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913672704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432897.320565:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432897.320568:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432897.320570:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432897.320573:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5266 00000100:00080000:0.0:1789432897.320577:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432897.320581:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432897.320590:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432897.320602:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913672832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432897.320611:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432897.320614:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432897.320615:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432897.320619:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5266 00000100:00080000:0.0:1789432897.320622:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432897.320626:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432897.320630:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432897.320640:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913672960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432897.320649:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432897.320652:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432897.320654:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432897.320657:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5266 00000100:00080000:0.0:1789432897.320659:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432897.320664:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432897.320668:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432897.320677:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913673088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432897.321242:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913672704:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789432897.321298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432897.321334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432897/real 1789432897] req@ffff9811c8cd6300 x1876350913672704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432897.321349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913672704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432897.321364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432897.321367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432897.321370:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913672704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432897.321377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432897.321396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432897.321408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432897/real 1789432897] req@ffff9811c8cd7b80 x1876350913672832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432897.321418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913672832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432897.321424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432897.321427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432897.321430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432897.321435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432897.321450:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432897.321458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432897/real 1789432897] req@ffff9811c8cd6a00 x1876350913672960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432897.321469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913672960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432897.321476:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432897.321478:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432897.321481:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913672960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432897.321487:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432897.321503:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432897.321511:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432897/real 1789432897] req@ffff9811c8cd4a80 x1876350913673088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432897.321521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913673088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432897.321527:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432897.321545:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432897.321547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432898.214020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432899.238044:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432900.262125:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432901.286144:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432902.310019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432902.437963:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432902.437981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432902.437987:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432902.437996:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5272 00000100:00080000:1.0:1789432902.438003:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432902.438014:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432902.438053:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913673216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432902.438080:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432902.438084:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432902.438085:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432902.438089:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5272 00000100:00080000:1.0:1789432902.438093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432902.438100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432902.438109:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432902.438115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913673216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432902.438123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913673344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432902.438135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432902.438138:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432902.438140:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432902.438143:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5272 00000100:00080000:1.0:1789432902.438146:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432902.438151:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432902.438155:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432902.438166:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913673472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432902.438176:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432902.438179:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432902.438181:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789432902.438198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789432902.438200:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5272 00000100:00080000:1.0:1789432902.438214:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432902.438220:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432902.438226:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432902.438239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432902/real 1789432902] req@ffff9811d0bdd180 x1876350913673216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432902.438241:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913673600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432902.438288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913673216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432902.438296:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432902.438299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432902.438302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913673216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432902.438321:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432902.438347:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432902.438359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432902/real 1789432902] req@ffff9811d0bdc380 x1876350913673344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432902.438371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913673344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432902.438379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432902.438382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432902.438385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432902.438391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432902.438408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432902.438417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432902/real 1789432902] req@ffff9811d0bddc00 x1876350913673472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432902.438426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913673472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432902.438433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432902.438436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432902.438438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432902.438444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673600:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432902.438462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432902.438471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432902/real 1789432902] req@ffff9811d0bdd500 x1876350913673600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432902.438482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913673600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432902.438489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432902.438502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432902.438507:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432903.334003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432904.358916:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432905.381995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432906.406011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432907.430015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432907.557979:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432907.557989:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432907.557993:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432907.558002:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5277 00000100:00080000:2.0:1789432907.558010:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432907.558021:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432907.558055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913673728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432907.558080:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432907.558084:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432907.558086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432907.558090:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5277 00000100:00080000:2.0:1789432907.558093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432907.558099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432907.558108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432907.558122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913673728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432907.558123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913673856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432907.558139:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432907.558143:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432907.558145:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432907.558149:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5277 00000100:00080000:2.0:1789432907.558153:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432907.558158:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432907.558163:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432907.558175:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913673984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432907.558186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432907.558186:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432907.558191:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432907.558193:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432907.558198:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5277 00000100:00080000:2.0:1789432907.558203:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432907.558208:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432907.558214:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432907.558229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432907/real 1789432907] req@ffff9811d2103b80 x1876350913673728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432907.558230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913674112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432907.558268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913673728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432907.558324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432907.558329:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432907.558332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913673728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432907.558352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432907.558376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432907.558389:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432907/real 1789432907] req@ffff9811d2101c00 x1876350913673856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432907.558408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913673856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432907.558424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432907.558427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432907.558430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432907.558436:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432907.558465:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432907.558478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432907/real 1789432907] req@ffff9811d2102a00 x1876350913673984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432907.558490:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913673984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432907.558497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432907.558500:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432907.558504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913673984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432907.558509:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432907.558541:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432907.558552:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432907/real 1789432907] req@ffff9811d2100e00 x1876350913674112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432907.558563:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913674112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432907.558578:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432907.558581:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432907.558584:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432908.454068:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432909.478061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432910.502050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432911.526017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432912.550012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432912.677961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432912.677971:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432912.677975:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432912.677984:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5282 00000100:00080000:0.0:1789432912.677993:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432912.678003:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432912.678039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913674240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432912.678068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432912.678071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432912.678073:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432912.678076:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5282 00000100:00080000:0.0:1789432912.678080:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432912.678084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432912.678093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432912.678103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913674368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432912.678113:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432912.678116:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432912.678118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432912.678121:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5282 00000100:00080000:0.0:1789432912.678123:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432912.678127:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432912.678130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432912.678137:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913674496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432912.678146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432912.678149:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432912.678151:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432912.678154:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5282 00000100:00080000:0.0:1789432912.678157:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432912.678160:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432912.678164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432912.678173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913674624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432912.678244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913674240:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789432912.678302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432912.678334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432912/real 1789432912] req@ffff9811c8cd5500 x1876350913674240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432912.678350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913674240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432912.678361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432912.678365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432912.678369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913674240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432912.678375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432912.678402:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432912.678413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432912/real 1789432912] req@ffff9811c8cd6d80 x1876350913674368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432912.678443:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913674368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432912.678451:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432912.678454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432912.678457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432912.678463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432912.678480:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432912.678488:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432912/real 1789432912] req@ffff9811c8cd5f80 x1876350913674496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432912.678506:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913674496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432912.678513:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432912.678516:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432912.678519:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432912.678524:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674624:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432912.678540:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432912.678548:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432912/real 1789432912] req@ffff9811c8cd4a80 x1876350913674624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432912.678558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913674624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432912.678566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432912.678569:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432912.678572:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432913.574005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432914.597956:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432915.622045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432916.645942:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432917.670019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432917.797979:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432917.797997:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432917.798002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432917.798012:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5287 00000100:00080000:0.0:1789432917.798019:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432917.798030:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432917.798063:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913674752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432917.798087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432917.798091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432917.798093:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432917.798097:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5287 00000100:00080000:0.0:1789432917.798101:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432917.798106:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432917.798116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432917.798120:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913674752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432917.798132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913674880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432917.798143:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432917.798146:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432917.798148:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432917.798152:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5287 00000100:00080000:0.0:1789432917.798155:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432917.798160:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432917.798165:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432917.798175:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913675008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432917.798190:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432917.798193:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432917.798194:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432917.798198:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5287 00000800:00000100:1.0:1789432917.798202:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789432917.798205:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432917.798211:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432917.798217:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432917.798230:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913675136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432917.798246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432917/real 1789432917] req@ffff9811c8cd6a00 x1876350913674752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432917.798271:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913674752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432917.798287:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432917.798290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432917.798294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913674752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432917.798323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432917.798344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432917.798356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432917/real 1789432917] req@ffff9811c8cd7b80 x1876350913674880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432917.798367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913674880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432917.798374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432917.798376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432917.798379:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913674880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432917.798398:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432917.798416:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432917.798426:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432917/real 1789432917] req@ffff9811c8cd6300 x1876350913675008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432917.798437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913675008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432917.798443:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432917.798446:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432917.798449:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432917.798455:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432917.798472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432917.798481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432917/real 1789432917] req@ffff9811c8cd4000 x1876350913675136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432917.798491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913675136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432917.798499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432917.798501:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432917.798504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432918.694003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432919.718014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432920.742014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432921.766029:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432922.790012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432922.917936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432922.917945:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432922.917949:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432922.917956:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5292 00000100:00080000:0.0:1789432922.917964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432922.917973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432922.918009:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913675264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432922.918032:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432922.918036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432922.918038:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432922.918042:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5292 00000100:00080000:0.0:1789432922.918046:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432922.918052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432922.918062:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432922.918066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913675264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432922.918078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913675392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432922.918089:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432922.918093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432922.918095:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432922.918099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5292 00000100:00080000:0.0:1789432922.918111:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432922.918116:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432922.918121:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789432922.918122:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432922.918135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913675520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432922.918143:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432922.918145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432922.918147:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432922.918150:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5292 00000100:00080000:0.0:1789432922.918153:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432922.918158:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432922.918159:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432922/real 1789432922] req@ffff9811c8cd5180 x1876350913675264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789432922.918161:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432922.918173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913675648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432922.918183:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913675264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432922.918196:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432922.918200:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432922.918203:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913675264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432922.918222:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432922.918240:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432922.918249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432922/real 1789432922] req@ffff9811c8cd7800 x1876350913675392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432922.918257:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913675392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432922.918263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432922.918266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432922.918268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432922.918273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432922.918290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432922.918300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432922/real 1789432922] req@ffff9811c8cd7480 x1876350913675520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432922.918311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913675520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432922.918323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432922.918325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432922.918328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432922.918334:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432922.918363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432922.918374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432922/real 1789432922] req@ffff9811c8cd7100 x1876350913675648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432922.918385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913675648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432922.918398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432922.918401:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432922.918403:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432923.814051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432924.837993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432925.862020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432926.886527:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432927.910063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432928.037981:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432928.037992:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432928.037997:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432928.038008:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5297 00000100:00080000:1.0:1789432928.038020:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432928.038036:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432928.038082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913675776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432928.038111:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432928.038113:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432928.038115:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432928.038118:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5297 00000100:00080000:1.0:1789432928.038121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432928.038126:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432928.038137:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432928.038148:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913675904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432928.038157:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913675776:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432928.038160:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432928.038166:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432928.038171:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432928.038178:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5297 00000100:00080000:1.0:1789432928.038181:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432928.038186:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432928.038191:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432928.038204:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913676032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432928.038215:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432928.038219:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432928.038221:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432928.038226:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5297 00000100:00080000:1.0:1789432928.038230:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432928.038234:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432928.038238:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432928.038248:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913676160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432928.038273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432928.038304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432928.038342:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432928.934010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432929.958010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432930.982045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789432931.110201:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789432931.110255:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789432931.110270:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432931.110323:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432931.110332:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789432931.110341:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789432931.110455:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432928/real 1789432931] req@ffff9811d0bdea00 x1876350913675904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432931.110478:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913675904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432931.110490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432931.110495:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432931.110499:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913675904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432931.110507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432928/real 1789432931] req@ffff9811d0bdfb80 x1876350913676032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432931.110519:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913676032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432931.110526:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432931.110529:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432931.110532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432931.110538:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432928/real 1789432931] req@ffff9811d0bdf480 x1876350913676160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432931.110553:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913676160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432931.110561:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432931.110564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432931.110567:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676160:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789432931.110573:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432928/real 1789432931] req@ffff9811d0bde680 x1876350913675776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432931.110583:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913675776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432931.110589:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432931.110592:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432931.110595:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913675776:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789432932.005976:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432933.030334:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432933.158095:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432933.158105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432933.158110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432933.158119:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5302 00000100:00080000:2.0:1789432933.158128:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432933.158139:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432933.158173:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913676288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432933.158203:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432933.158206:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432933.158208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432933.158212:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5302 00000100:00080000:2.0:1789432933.158215:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432933.158220:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432933.158228:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432933.158239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913676416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432933.158248:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432933.158251:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432933.158253:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432933.158256:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5302 00000100:00080000:2.0:1789432933.158259:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432933.158263:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432933.158267:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432933.158287:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913676544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432933.158303:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432933.158306:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432933.158308:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432933.158311:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5302 00000100:00080000:2.0:1789432933.158314:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432933.158319:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432933.158323:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432933.158333:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913676672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432933.158400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913676288:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789432933.158465:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432933.158503:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432933/real 1789432933] req@ffff9811d2101880 x1876350913676288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432933.158519:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913676288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432933.158528:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432933.158532:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432933.158535:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913676288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432933.158542:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432933.158561:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432933.158572:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432933/real 1789432933] req@ffff9811d2101180 x1876350913676416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432933.158583:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913676416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432933.158591:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432933.158594:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432933.158597:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432933.158602:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432933.158618:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432933.158628:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432933/real 1789432933] req@ffff9811d2101f80 x1876350913676544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432933.158640:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913676544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432933.158647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432933.158650:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432933.158653:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432933.158658:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432933.158682:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432933.158692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432933/real 1789432933] req@ffff9811d2100e00 x1876350913676672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432933.158703:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913676672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432933.158710:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432933.158713:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432933.158716:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432934.054443:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432935.078107:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432936.102039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432937.126031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432938.150126:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432938.278011:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432938.278019:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432938.278023:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432938.278032:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5307 00000100:00080000:1.0:1789432938.278041:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432938.278052:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432938.278098:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913676800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432938.278132:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432938.278136:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432938.278138:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432938.278146:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5307 00000100:00080000:1.0:1789432938.278150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432938.278154:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432938.278163:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432938.278176:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913676928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432938.278186:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432938.278189:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432938.278190:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432938.278193:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5307 00000100:00080000:1.0:1789432938.278197:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432938.278202:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432938.278205:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432938.278213:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913677056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432938.278221:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789432938.278225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913676800:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432938.278226:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432938.278230:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432938.278247:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5307 00000100:00080000:1.0:1789432938.278252:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432938.278256:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432938.278261:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432938.278273:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913677184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432938.278310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432938.278342:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432938/real 1789432938] req@ffff9811d0bde680 x1876350913676800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432938.278354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913676800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432938.278362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432938.278364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432938.278366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913676800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432938.278372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432938.278386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432938.278393:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432938/real 1789432938] req@ffff9811d0bdf480 x1876350913676928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432938.278401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913676928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432938.278405:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432938.278407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432938.278409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913676928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432938.278441:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432938.278460:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432938.278470:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432938/real 1789432938] req@ffff9811d0bdfb80 x1876350913677056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432938.278482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913677056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432938.278490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432938.278497:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432938.278500:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432938.278507:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677184:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432938.278526:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432938.278536:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432938/real 1789432938] req@ffff9811d0bdea00 x1876350913677184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432938.278548:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913677184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432938.278555:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432938.278558:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432938.278561:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432939.174022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432940.197989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432941.222387:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432942.246490:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432943.270007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432943.399363:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432943.399374:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432943.399378:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432943.399399:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5312 00000100:00080000:0.0:1789432943.399408:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432943.399419:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432943.399455:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913677312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432943.399481:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432943.399484:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432943.399486:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432943.399490:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5312 00000100:00080000:0.0:1789432943.399503:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432943.399509:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432943.399518:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432943.399532:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913677440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432943.399550:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432943.399553:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432943.399555:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432943.399558:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5312 00000100:00080000:0.0:1789432943.399561:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432943.399566:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432943.399571:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432943.399575:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913677312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432943.399585:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913677568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432943.399597:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432943.399605:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432943.399606:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432943.399610:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5312 00000100:00080000:0.0:1789432943.399613:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432943.399618:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432943.399622:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432943.399634:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913677696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432943.399694:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432943.399728:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432943/real 1789432943] req@ffff9811c8cd5c00 x1876350913677312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432943.399745:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913677312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432943.399756:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432943.399759:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432943.399763:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913677312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432943.399784:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432943.399991:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432943.400005:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432943/real 1789432943] req@ffff9811c8cd4700 x1876350913677440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432943.400018:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913677440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432943.400027:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432943.400031:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432943.400035:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432943.400041:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432943.400059:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432943.400081:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432943/real 1789432943] req@ffff9811c8cd6680 x1876350913677568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432943.400093:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913677568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432943.400101:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432943.400104:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432943.400107:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432943.400113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432943.400129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432943.400138:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432943/real 1789432943] req@ffff9811c8cd7100 x1876350913677696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432943.400150:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913677696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432943.400158:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432943.400161:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432943.400164:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432944.294134:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432945.318036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432946.344363:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432947.366035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432948.390674:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432948.517955:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432948.517966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432948.517969:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432948.517979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5318 00000100:00080000:2.0:1789432948.517988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432948.517999:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432948.518038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913677824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432948.518066:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432948.518069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432948.518072:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432948.518077:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5318 00000100:00080000:2.0:1789432948.518080:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432948.518085:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432948.518094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432948.518108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913677952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432948.518119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913677824:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432948.518120:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432948.518128:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432948.518130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432948.518134:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5318 00000100:00080000:2.0:1789432948.518138:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432948.518143:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432948.518147:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432948.518177:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913678080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789432948.518195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432948.518198:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432948.518202:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432948.518205:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432948.518209:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5318 00000100:00080000:2.0:1789432948.518212:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432948.518217:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432948.518227:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432948.518243:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432948/real 1789432948] req@ffff9811d2102a00 x1876350913677824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789432948.518243:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913678208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432948.518282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913677824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432948.518293:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432948.518298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432948.518302:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913677824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432948.518323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432948.518345:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432948.518357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432948/real 1789432948] req@ffff9811d2101c00 x1876350913677952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432948.518367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913677952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432948.518374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432948.518377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432948.518380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913677952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432948.518385:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432948.518401:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432948.518429:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432948/real 1789432948] req@ffff9811d2103b80 x1876350913678080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432948.518438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913678080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432948.518450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432948.518454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432948.518457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432948.518462:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432948.518482:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432948.518501:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432948/real 1789432948] req@ffff9811d2101500 x1876350913678208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432948.518513:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913678208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432948.518518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432948.518520:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432948.518522:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432949.414054:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432950.438087:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432951.462321:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432952.485989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432953.510540:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432953.637990:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432953.638000:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432953.638004:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432953.638014:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5323 00000100:00080000:0.0:1789432953.638023:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432953.638031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432953.638062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913678336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432953.638084:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432953.638087:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432953.638089:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432953.638093:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5323 00000100:00080000:0.0:1789432953.638097:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432953.638104:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432953.638114:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432953.638131:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913678336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432953.638142:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913678464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432953.638154:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432953.638157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432953.638160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432953.638163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5323 00000100:00080000:0.0:1789432953.638182:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432953.638188:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432953.638194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789432953.638195:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789432953.638211:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913678592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432953.638222:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432953.638225:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432953.638227:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432953.638231:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5323 00000100:00080000:0.0:1789432953.638234:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789432953.638237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432953/real 1789432953] req@ffff9811c8cd7480 x1876350913678336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432953.638251:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432953.638258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913678336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789432953.638259:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789432953.638273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789432953.638278:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913678720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432953.638281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432953.638290:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913678336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432953.638318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432953.638341:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432953.638352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432953/real 1789432953] req@ffff9811c8cd7800 x1876350913678464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432953.638360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913678464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432953.638365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432953.638367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432953.638369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432953.638374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432953.638391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432953.638401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432953/real 1789432953] req@ffff9811c8cd5180 x1876350913678592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432953.638412:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913678592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432953.638420:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432953.638423:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432953.638426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432953.638432:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678720:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432953.638451:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432953.638461:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432953/real 1789432953] req@ffff9811c8cd4000 x1876350913678720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432953.638473:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913678720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432953.638480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432953.638483:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432953.638486:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432954.534007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432955.558024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432956.582075:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432957.605989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432958.631041:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432958.757923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432958.757931:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432958.757934:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432958.757942:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5328 00000100:00080000:1.0:1789432958.757949:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432958.757957:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432958.757986:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913678848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432958.758007:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432958.758009:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432958.758011:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432958.758013:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5328 00000100:00080000:1.0:1789432958.758015:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432958.758017:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432958.758024:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432958.758033:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913678976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432958.758044:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432958.758046:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432958.758047:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432958.758050:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5328 00000100:00080000:1.0:1789432958.758053:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789432958.758054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913678848:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432958.758057:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432958.758062:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432958.758068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913679104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432958.758073:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432958.758075:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432958.758077:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432958.758079:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5328 00000100:00080000:1.0:1789432958.758081:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432958.758083:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432958.758099:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432958.758115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913679232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789432958.758129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432958.758166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432958/real 1789432958] req@ffff9811d0bded80 x1876350913678848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432958.758180:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913678848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432958.758187:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432958.758189:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432958.758192:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913678848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432958.758209:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432958.758224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432958.758232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432958/real 1789432958] req@ffff9811d0bdc000 x1876350913678976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432958.758243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913678976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432958.758250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432958.758253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432958.758256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913678976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432958.758262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432958.758280:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432958.758289:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432958/real 1789432958] req@ffff9811d0bdc700 x1876350913679104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432958.758300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913679104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432958.758305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432958.758306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432958.758308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432958.758312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432958.758324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432958.758331:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432958/real 1789432958] req@ffff9811d0bdd880 x1876350913679232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432958.758341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913679232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432958.758349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432958.758352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432958.758355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432959.654018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432960.678009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432961.702014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432962.726006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432963.750017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789432963.877945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432963.877955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789432963.877959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432963.877968:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5333 00000100:00080000:0.0:1789432963.877977:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432963.877989:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789432963.878022:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913679360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432963.878059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432963.878062:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789432963.878063:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432963.878067:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5333 00000100:00080000:0.0:1789432963.878070:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432963.878075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432963.878085:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432963.878087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913679360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432963.878100:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913679488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432963.878112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432963.878116:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789432963.878118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432963.878122:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5333 00000100:00080000:0.0:1789432963.878135:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432963.878140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432963.878144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789432963.878148:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432963.878155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913679616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789432963.878163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789432963.878165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789432963.878166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789432963.878169:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5333 00000100:00080000:0.0:1789432963.878171:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789432963.878174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789432963.878178:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789432963.878189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913679744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432963.878204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432963/real 1789432963] req@ffff9811c8cd6300 x1876350913679360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432963.878228:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913679360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432963.878238:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432963.878241:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432963.878245:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913679360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432963.878264:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432963.878286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432963.878299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432963/real 1789432963] req@ffff9811c8cd7b80 x1876350913679488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432963.878311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913679488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432963.878319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432963.878323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432963.878326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432963.878331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432963.878349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432963.878359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432963/real 1789432963] req@ffff9811c8cd6a00 x1876350913679616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432963.878370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913679616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432963.878383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432963.878387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432963.878390:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432963.878396:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432963.878413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432963.878422:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432963/real 1789432963] req@ffff9811c8cd4a80 x1876350913679744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432963.878433:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913679744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432963.878441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432963.878444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432963.878447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913679744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432964.773978:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432965.798066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432966.822005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432967.846015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432968.869965:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432968.997945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432968.997955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432968.997959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432968.997967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5338 00000100:00080000:2.0:1789432968.997975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432968.997985:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432968.998028:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913679872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432968.998053:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432968.998056:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432968.998058:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432968.998063:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5338 00000100:00080000:2.0:1789432968.998066:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432968.998071:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432968.998080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432968.998095:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913679872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789432968.998095:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913680000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432968.998117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432968.998119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432968.998121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432968.998125:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5338 00000100:00080000:2.0:1789432968.998128:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432968.998132:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432968.998138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432968.998150:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913680128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432968.998158:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432968.998162:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432968.998166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789432968.998167:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789432968.998174:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5338 00000100:00080000:2.0:1789432968.998178:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432968.998184:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432968.998190:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432968.998201:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913680256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432968.998208:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432968/real 1789432968] req@ffff9811d2102d80 x1876350913679872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432968.998226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913679872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432968.998233:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432968.998236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432968.998239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913679872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432968.998255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432968.998271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432968.998278:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432968/real 1789432968] req@ffff9811d2103800 x1876350913680000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432968.998285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913680000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432968.998290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432968.998292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432968.998294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432968.998297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432968.998305:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432968.998310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432968/real 1789432968] req@ffff9811d2100000 x1876350913680128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432968.998319:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913680128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432968.998326:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432968.998328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432968.998331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432968.998336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680256:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432968.998355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432968.998365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432968/real 1789432968] req@ffff9811d2101500 x1876350913680256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432968.998376:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913680256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432968.998381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432968.998383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432968.998385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432969.893991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432970.917973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432971.942020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432972.966066:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432973.990495:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432974.117888:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432974.117898:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432974.117903:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432974.117913:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5343 00000100:00080000:2.0:1789432974.117921:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432974.117932:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432974.117962:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913680384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432974.117991:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432974.117993:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432974.117995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432974.117998:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5343 00000100:00080000:2.0:1789432974.118000:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432974.118004:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432974.118012:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432974.118021:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913680512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432974.118028:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913680384:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432974.118030:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432974.118035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432974.118038:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432974.118041:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5343 00000100:00080000:2.0:1789432974.118043:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432974.118045:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432974.118047:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432974.118055:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913680640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432974.118066:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432974.118069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432974.118071:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432974.118074:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5343 00000100:00080000:2.0:1789432974.118079:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432974.118084:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432974.118089:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789432974.118094:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432974.118103:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913680768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432974.118143:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432974/real 1789432974] req@ffff9811d2103b80 x1876350913680384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432974.118161:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913680384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432974.118171:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432974.118174:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432974.118178:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913680384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432974.118198:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432974.118218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432974.118242:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432974/real 1789432974] req@ffff9811d2101c00 x1876350913680512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432974.118528:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913680512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432974.118543:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432974.118546:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432974.118549:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432974.118555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432974.118578:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432974.118590:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432974/real 1789432974] req@ffff9811d2102a00 x1876350913680640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432974.118601:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913680640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432974.118609:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432974.118612:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432974.118615:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432974.118621:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432974.118640:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432974.118650:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432974/real 1789432974] req@ffff9811d2100e00 x1876350913680768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432974.118661:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913680768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789432999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432974.118669:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432974.118672:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432974.118676:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913680768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432975.014011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432976.038329:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432977.062066:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432978.086122:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432979.110272:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432979.237998:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432979.238021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432979.238027:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432979.238040:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5348 00000100:00080000:2.0:1789432979.238052:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432979.238068:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432979.238117:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913680896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432979.238147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432979.238154:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432979.238156:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432979.238161:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5348 00000100:00080000:2.0:1789432979.238165:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432979.238172:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432979.238184:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432979.238200:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913681024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432979.238205:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913680896:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432979.238212:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432979.238215:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432979.238217:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432979.238220:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5348 00000100:00080000:2.0:1789432979.238223:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432979.238226:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432979.238231:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432979.238250:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913681152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432979.238261:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432979.238264:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432979.238267:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432979.238271:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5348 00000100:00080000:2.0:1789432979.238275:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432979.238280:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789432979.238286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789432979.238289:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432979.238305:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913681280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789432979.238348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432979/real 1789432979] req@ffff9811d2101f80 x1876350913680896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432979.238368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913680896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432979.238379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432979.238384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432979.238387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913680896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789432979.238413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432979.238434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432979.238447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432979/real 1789432979] req@ffff9811d2101180 x1876350913681024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432979.238458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913681024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432979.238467:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432979.238470:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432979.238473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432979.238478:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789432979.238493:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432979.238503:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432979/real 1789432979] req@ffff9811d2101880 x1876350913681152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432979.238514:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913681152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432979.238521:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432979.238524:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432979.238527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789432979.238532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681280:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789432979.238551:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432979.238560:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432979/real 1789432979] req@ffff9811d2100e00 x1876350913681280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789432979.238572:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913681280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789432979.238580:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789432979.238583:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789432979.238586:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432980.134027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432981.160209:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432982.182022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432983.206030:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432984.230034:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432984.358282:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432984.358290:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432984.358293:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432984.358301:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5353 00000100:00080000:1.0:1789432984.358309:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432984.358318:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432984.358362:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913681408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432984.358390:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432984.358394:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432984.358396:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432984.358400:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5353 00000100:00080000:1.0:1789432984.358404:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432984.358409:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432984.358420:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432984.358433:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913681536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432984.358444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913681408:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432984.358446:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432984.358453:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432984.358457:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432984.358462:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5353 00000100:00080000:1.0:1789432984.358465:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432984.358469:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432984.358473:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432984.358482:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913681664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432984.358492:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432984.358496:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432984.358498:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432984.358502:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5353 00000100:00080000:1.0:1789432984.358506:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432984.358510:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432984.358515:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789432984.358517:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789432984.358533:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913681792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432984.358583:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432984/real 1789432984] req@ffff9811d0bdf800 x1876350913681408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432984.358602:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913681408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432984.358612:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432984.358616:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432984.358620:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913681408:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432984.358641:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432984.358662:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432984.358673:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432984/real 1789432984] req@ffff9811d0bde300 x1876350913681536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432984.358684:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913681536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432984.358692:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432984.358695:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432984.358698:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432984.358703:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432984.358721:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432984.358729:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432984/real 1789432984] req@ffff9811d0bdf100 x1876350913681664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432984.358751:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913681664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432984.358758:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432984.358761:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432984.358764:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432984.358773:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681792:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432984.358791:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432984.358801:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432984/real 1789432984] req@ffff9811d0bdd880 x1876350913681792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432984.358816:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913681792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432984.358823:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432984.358826:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432984.358829:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913681792:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789432985.254155:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432986.277981:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432987.302041:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432988.326076:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432989.350022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432989.477938:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432989.477947:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432989.477952:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432989.477960:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5359 00000100:00080000:2.0:1789432989.477969:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432989.477980:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432989.478011:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913681920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432989.478032:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432989.478035:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432989.478036:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432989.478039:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5359 00000100:00080000:2.0:1789432989.478041:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432989.478044:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432989.478052:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432989.478059:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913682048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432989.478066:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432989.478068:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432989.478070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432989.478073:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5359 00000100:00080000:2.0:1789432989.478076:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789432989.478081:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913681920:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789432989.478084:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432989.478092:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432989.478104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913682176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432989.478115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432989.478119:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432989.478121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432989.478124:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5359 00000100:00080000:2.0:1789432989.478129:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432989.478135:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432989.478140:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789432989.478145:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789432989.478160:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913682304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432989.478191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432989/real 1789432989] req@ffff9811d2102a00 x1876350913681920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432989.478219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913681920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432989.478229:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432989.478233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432989.478236:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913681920:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432989.478243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432989.478263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432989.478273:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432989/real 1789432989] req@ffff9811d2101c00 x1876350913682048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432989.478284:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913682048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432989.478290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432989.478293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432989.478296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432989.478315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432989.478332:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432989.478341:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432989/real 1789432989] req@ffff9811d2103b80 x1876350913682176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432989.478351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913682176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432989.478357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432989.478360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432989.478363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432989.478368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682304:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432989.478383:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432989.478391:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432989/real 1789432989] req@ffff9811d2101500 x1876350913682304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432989.478401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913682304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432989.478408:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432989.478410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432989.478425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682304:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789432990.374011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432991.398009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789432992.422018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432993.446020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432994.470094:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789432994.597997:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432994.598011:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789432994.598017:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432994.598037:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5364 00000100:00080000:1.0:1789432994.598057:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432994.598073:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789432994.598133:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913682432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432994.598168:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432994.598172:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789432994.598174:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432994.598179:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5364 00000100:00080000:1.0:1789432994.598183:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432994.598189:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432994.598201:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432994.598217:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913682560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432994.598230:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789432994.598231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913682432:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789432994.598239:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789432994.598242:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432994.598246:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5364 00000100:00080000:1.0:1789432994.598250:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432994.598255:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432994.598260:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789432994.598274:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913682688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789432994.598285:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789432994.598289:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789432994.598292:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789432994.598296:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5364 00000100:00080000:1.0:1789432994.598300:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789432994.598305:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789432994.598310:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789432994.598322:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913682816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432994.598379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432994.598409:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432994.598426:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682816:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789432995.494005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432996.517948:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789432997.542033:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789432997.671498:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789432997.671569:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789432997.671591:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432997.671646:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432997.671658:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789432997.671666:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789432997.671936:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432994/real 1789432997] req@ffff9811d0bdc000 x1876350913682560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432997.671969:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913682560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432997.671987:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432997.671993:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432997.671997:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432997.672006:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432994/real 1789432997] req@ffff9811d0bded80 x1876350913682688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432997.672018:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913682688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432997.672026:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432997.672030:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432997.672033:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432997.672041:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432994/real 1789432997] req@ffff9811d0bdea00 x1876350913682816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432997.672063:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913682816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432997.672071:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432997.672075:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432997.672078:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913682816:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789432997.672085:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432994/real 1789432997] req@ffff9811d0bdc700 x1876350913682432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432997.672103:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913682432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432997.672111:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432997.672116:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432997.672119:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913682432:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789432998.566006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789432999.590123:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789432999.718056:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432999.718067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789432999.718070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432999.718081:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5369 00000100:00080000:2.0:1789432999.718089:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432999.718100:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789432999.718140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913682944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432999.718181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432999.718185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789432999.718187:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432999.718192:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5369 00000100:00080000:2.0:1789432999.718196:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432999.718201:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432999.718213:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432999.718226:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913683072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432999.718235:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432999.718239:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789432999.718241:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432999.718244:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5369 00000100:00080000:2.0:1789432999.718248:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432999.718252:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432999.718257:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789432999.718268:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913683200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789432999.718278:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789432999.718281:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789432999.718283:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789432999.718299:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5369 00000100:00080000:2.0:1789432999.718304:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789432999.718309:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789432999.718313:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789432999.718325:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913683328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789432999.718429:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913682944:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789432999.718517:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432999.718576:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432999/real 1789432999] req@ffff9811d2100000 x1876350913682944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432999.718596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913682944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432999.718608:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432999.718614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432999.718618:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913682944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789432999.718639:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432999.718664:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432999.718677:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432999/real 1789432999] req@ffff9811d2103800 x1876350913683072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432999.719888:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913683072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432999.719901:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432999.719907:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432999.719911:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432999.719922:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789432999.719973:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432999.719992:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432999/real 1789432999] req@ffff9811d2102d80 x1876350913683200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432999.720003:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913683200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432999.720012:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432999.720015:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432999.720018:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789432999.720024:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683328:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789432999.720055:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789432999.720066:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789432999/real 1789432999] req@ffff9811d2101500 x1876350913683328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789432999.720077:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913683328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789432999.720085:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789432999.720088:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789432999.720091:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683328:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433000.614071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433001.638124:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433002.662616:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433003.686008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433004.710071:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433004.838739:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433004.838749:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433004.838762:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433004.838771:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5374 00000100:00080000:2.0:1789433004.838780:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433004.838791:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433004.838827:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913683456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433004.838870:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433004.838875:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433004.838876:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433004.838881:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5374 00000100:00080000:2.0:1789433004.838884:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433004.838890:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433004.838899:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433004.838913:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913683584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433004.838921:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913683456:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433004.838937:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433004.838940:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433004.838942:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433004.838946:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5374 00000100:00080000:2.0:1789433004.838949:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433004.838954:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433004.838959:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433004.838970:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913683712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433004.838980:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433004.838984:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433004.838985:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433004.838988:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5374 00000100:00080000:2.0:1789433004.838991:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433004.838995:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433004.838999:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433004.839011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913683840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433004.839057:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433004.839094:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433004/real 1789433004] req@ffff9811d2103b80 x1876350913683456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433004.839115:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913683456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433004.839125:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433004.839129:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433004.839133:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913683456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433004.839153:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433004.839175:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433004.839187:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433004/real 1789433004] req@ffff9811d2101c00 x1876350913683584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433004.839198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913683584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433004.839205:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433004.839208:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433004.839211:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433004.839216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433004.839233:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433004.839241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433004/real 1789433004] req@ffff9811d2102a00 x1876350913683712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433004.839252:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913683712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433004.839264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433004.839267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433004.839269:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433004.839274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683840:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433004.839292:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433004.839301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433004/real 1789433004] req@ffff9811d2100e00 x1876350913683840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433004.839311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913683840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433004.839318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433004.839321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433004.839324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913683840:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433005.733998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433006.758443:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433007.782140:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433008.806232:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433009.830020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433009.957981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433009.957990:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433009.957994:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433009.958003:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5379 00000100:00080000:3.0:1789433009.958011:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433009.958021:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433009.958056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913683968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433009.958086:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433009.958089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433009.958091:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433009.958096:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5379 00000100:00080000:3.0:1789433009.958099:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433009.958104:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433009.958113:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433009.958126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913684096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433009.958135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433009.958138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433009.958140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433009.958143:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5379 00000100:00080000:3.0:1789433009.958146:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433009.958151:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433009.958155:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433009.958164:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913684224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433009.958174:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433009.958177:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433009.958179:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433009.958183:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5379 00000100:00080000:3.0:1789433009.958185:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433009.958189:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433009.958193:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433009.958201:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913684352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433009.958325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913683968:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433009.958380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433009.958414:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433009/real 1789433009] req@ffff9811d1bc6a00 x1876350913683968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433009.958427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913683968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433009.958434:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433009.958436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433009.958439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913683968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433009.958444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433009.958465:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433009.958476:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433009/real 1789433009] req@ffff9811d1bc7480 x1876350913684096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433009.958488:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913684096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433009.958496:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433009.958499:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433009.958502:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433009.958506:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433009.958524:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433009.958535:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433009/real 1789433009] req@ffff9811d1bc7800 x1876350913684224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433009.958546:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913684224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433009.958554:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433009.958557:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433009.958561:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433009.958566:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684352:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433009.958583:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433009.958590:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433009/real 1789433009] req@ffff9811d1bc5c00 x1876350913684352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433009.958597:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913684352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433009.958602:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433009.958604:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433009.958606:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684352:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433010.854794:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433011.878545:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433012.904020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433013.926221:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433014.950005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433015.077946:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433015.077956:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433015.077960:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433015.077971:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5384 00000100:00080000:3.0:1789433015.077979:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433015.079671:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433015.079717:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913684480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433015.079739:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433015.079743:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433015.079745:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433015.079750:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5384 00000100:00080000:3.0:1789433015.079755:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433015.079762:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433015.079771:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433015.079784:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913684608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433015.079793:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433015.079796:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433015.079798:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433015.079801:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5384 00000100:00080000:3.0:1789433015.079804:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433015.079808:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433015.079812:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433015.079822:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913684736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433015.079856:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433015.079860:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433015.079862:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433015.079866:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5384 00000100:00080000:3.0:1789433015.079869:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433015.079873:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433015.079878:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433015.079889:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913684864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433015.079991:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913684480:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433015.080046:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433015.080086:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433015/real 1789433015] req@ffff9811d1bc5180 x1876350913684480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433015.080102:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913684480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433015.080112:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433015.080116:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433015.080120:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913684480:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433015.080127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433015.080146:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433015.080155:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433015/real 1789433015] req@ffff9811d1bc4000 x1876350913684608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433015.080173:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913684608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433015.080182:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433015.080185:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433015.080188:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433015.080193:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433015.080209:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433015.080218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433015/real 1789433015] req@ffff9811d1bc5f80 x1876350913684736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433015.080228:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913684736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433015.080236:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433015.080239:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433015.080242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433015.080248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684864:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433015.080267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433015.080276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433015/real 1789433015] req@ffff9811d1bc5880 x1876350913684864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433015.080287:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913684864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433015.080300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433015.080302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433015.080305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913684864:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433015.974629:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433016.998085:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433018.022014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433019.046726:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433020.070025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433020.197958:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433020.197965:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433020.197969:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433020.197976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5389 00000100:00080000:2.0:1789433020.197982:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433020.197992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433020.198025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913684992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433020.198051:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433020.198054:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433020.198055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433020.198058:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5389 00000100:00080000:2.0:1789433020.198061:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433020.198065:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433020.198072:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433020.198084:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913685120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433020.198094:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433020.198099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433020.198101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433020.198104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913684992:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433020.198105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5389 00000100:00080000:2.0:1789433020.198112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433020.198117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433020.198122:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433020.198149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913685248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433020.198160:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433020.198164:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433020.198166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433020.198168:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433020.198172:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5389 00000100:00080000:2.0:1789433020.198177:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433020.198193:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433020.198199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433020.198209:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433020/real 1789433020] req@ffff9811d2101880 x1876350913684992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789433020.198215:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913685376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433020.198235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913684992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433020.198247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433020.198251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433020.198254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913684992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433020.198271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433020.198287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433020.198295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433020/real 1789433020] req@ffff9811d2101180 x1876350913685120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433020.198302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913685120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433020.198307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433020.198309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433020.198311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433020.198314:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433020.198323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433020.198328:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433020/real 1789433020] req@ffff9811d2101f80 x1876350913685248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433020.198334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913685248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433020.198338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433020.198340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433020.198341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433020.198344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685376:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433020.198353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433020.198358:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433020/real 1789433020] req@ffff9811d2100e00 x1876350913685376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433020.198367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913685376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433020.198374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433020.198377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433020.198380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685376:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433021.094003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433022.118065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433023.141994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433024.166025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433025.189992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433025.317950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433025.317960:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433025.317964:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433025.317978:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5394 00000100:00080000:3.0:1789433025.317988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433025.317998:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433025.318037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913685504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433025.318062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433025.318066:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433025.318067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433025.318071:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5394 00000100:00080000:3.0:1789433025.318074:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433025.318079:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433025.318088:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433025.318099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913685632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433025.318111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913685504:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433025.318111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433025.318120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433025.318122:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433025.318126:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5394 00000100:00080000:3.0:1789433025.318129:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433025.318133:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433025.318137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433025.318148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913685760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433025.318157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433025.318161:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433025.318162:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433025.318166:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5394 00000100:00080000:3.0:1789433025.318168:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433025.318172:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433025.318177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433025.318178:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433025.318191:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913685888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433025.318220:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433025/real 1789433025] req@ffff9811d1bc6300 x1876350913685504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433025.318237:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913685504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433025.318248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433025.318253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433025.318256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913685504:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433025.318275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433025.318297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433025.318310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433025/real 1789433025] req@ffff9811d1bc7100 x1876350913685632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433025.318322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913685632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433025.318328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433025.318332:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433025.318335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433025.318340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433025.318360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433025.318371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433025/real 1789433025] req@ffff9811d1bc6680 x1876350913685760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433025.318383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913685760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433025.318391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433025.318393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433025.318396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433025.318402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685888:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433025.318422:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433025.318433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433025/real 1789433025] req@ffff9811d1bc5500 x1876350913685888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433025.318443:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913685888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433025.318449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433025.318450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433025.318452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913685888:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433026.214027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433027.238238:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433028.262026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433029.286027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433030.310012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433030.437964:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433030.437974:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433030.437977:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433030.437987:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5400 00000100:00080000:1.0:1789433030.437995:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433030.438004:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433030.438038:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913686016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433030.438064:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433030.438067:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433030.438069:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433030.438072:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5400 00000100:00080000:1.0:1789433030.438075:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433030.438080:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433030.438089:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433030.438101:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913686144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433030.438104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913686016:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433030.438118:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433030.438122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433030.438123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433030.438127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5400 00000100:00080000:1.0:1789433030.438130:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433030.438134:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433030.438138:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433030.438148:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913686272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433030.438158:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433030.438161:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433030.438163:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433030.438168:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5400 00000800:00000100:0.0:1789433030.438173:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433030.438174:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433030.438181:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433030.438186:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433030.438198:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913686400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433030.438212:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433030/real 1789433030] req@ffff9811d0bdfb80 x1876350913686016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433030.438230:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913686016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433030.438241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433030.438245:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433030.438248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913686016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433030.438269:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433030.438290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433030.438302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433030/real 1789433030] req@ffff9811d0bdf480 x1876350913686144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433030.438314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913686144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433030.438322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433030.438326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433030.438329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433030.438335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433030.438350:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433030.438357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433030/real 1789433030] req@ffff9811d0bde680 x1876350913686272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433030.438363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913686272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433030.438368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433030.438370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433030.438372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433030.438376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686400:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433030.438386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433030.438391:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433030/real 1789433030] req@ffff9811d0bdd500 x1876350913686400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433030.438397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913686400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433030.438402:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433030.438403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433030.438405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686400:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433031.334040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433032.358058:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433033.382101:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433034.406124:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433035.430154:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433035.557938:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433035.557948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433035.557952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433035.557962:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5405 00000100:00080000:0.0:1789433035.557970:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433035.557979:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433035.558012:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913686528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433035.558037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433035.558040:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433035.558042:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433035.558046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5405 00000100:00080000:0.0:1789433035.558050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433035.558055:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433035.558065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433035.558071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913686528:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433035.558078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913686656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433035.558089:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433035.558093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433035.558094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433035.558099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5405 00000100:00080000:0.0:1789433035.558102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433035.558115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433035.558119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789433035.558125:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433035.558131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913686784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433035.558142:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433035.558145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433035.558146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433035.558150:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5405 00000100:00100000:1.0:1789433035.558164:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433035/real 1789433035] req@ffff9811c8cd5f80 x1876350913686528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433035.558165:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433035.558174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433035.558181:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433035.558188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913686528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433035.558203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789433035.558205:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913686912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433035.558209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433035.558217:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913686528:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433035.558246:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433035.558270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433035.558282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433035/real 1789433035] req@ffff9811c8cd6d80 x1876350913686656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433035.558295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913686656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433035.558304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433035.558307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433035.558311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433035.558316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433035.558336:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433035.558346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433035/real 1789433035] req@ffff9811c8cd5500 x1876350913686784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433035.558356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913686784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433035.558364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433035.558367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433035.558370:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433035.558375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686912:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433035.558393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433035.558402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433035/real 1789433035] req@ffff9811c8cd4a80 x1876350913686912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433035.558411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913686912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433035.558422:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433035.558424:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433035.558426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913686912:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433036.454028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433037.478054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433038.502020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433039.526112:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433040.550058:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433040.678002:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433040.678011:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433040.678015:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433040.678024:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5410 00000100:00080000:1.0:1789433040.678033:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433040.678044:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433040.678081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913687040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433040.678119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433040.678124:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433040.678127:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433040.678135:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5410 00000100:00080000:1.0:1789433040.678140:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433040.678147:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433040.678163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433040.678165:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913687040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433040.678183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913687168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433040.678195:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433040.678199:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433040.678206:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433040.678209:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5410 00000100:00080000:1.0:1789433040.678213:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433040.678219:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433040.678224:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789433040.678227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433040.678238:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913687296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433040.678247:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433040.678249:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433040.678251:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433040.678254:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5410 00000100:00080000:1.0:1789433040.678258:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433040.678263:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433040.678264:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433040/real 1789433040] req@ffff9811d0bddc00 x1876350913687040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789433040.678267:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433040.678278:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913687424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433040.678295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913687040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433040.678307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433040.678312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433040.678331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913687040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433040.678356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433040.678380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433040.678392:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433040/real 1789433040] req@ffff9811d0bdc380 x1876350913687168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433040.678401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913687168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433040.678406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433040.678409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433040.678412:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433040.678418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433040.678439:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433040.678449:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433040/real 1789433040] req@ffff9811d0bdd180 x1876350913687296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433040.678460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913687296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433040.678467:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433040.678470:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433040.678473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433040.678479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687424:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433040.678513:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433040.678524:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433040/real 1789433040] req@ffff9811d0bdd500 x1876350913687424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433040.678535:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913687424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433040.678542:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433040.678546:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433040.678549:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687424:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433041.573980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433042.598009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433043.622006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433044.648049:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433045.670005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433045.797996:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433045.798006:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433045.798010:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433045.798020:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5415 00000100:00080000:2.0:1789433045.798030:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433045.798041:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433045.798095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913687552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433045.798127:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433045.798130:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433045.798132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433045.798136:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5415 00000100:00080000:2.0:1789433045.798140:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433045.798144:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433045.798153:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433045.798165:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913687680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433045.798173:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433045.798175:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433045.798177:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433045.798180:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5415 00000100:00080000:2.0:1789433045.798184:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433045.798189:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433045.798193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433045.798204:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913687552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433045.798205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913687808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433045.798221:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433045.798224:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433045.798226:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433045.798230:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5415 00000100:00080000:2.0:1789433045.798242:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433045.798247:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433045.798252:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433045.798264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433045.798267:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913687936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433045.798300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433045/real 1789433045] req@ffff9811d2102a00 x1876350913687552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433045.798338:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913687552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433045.798346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433045.798351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433045.798355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913687552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433045.798362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433045.798383:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433045.798400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433045/real 1789433045] req@ffff9811d2101c00 x1876350913687680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433045.798412:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913687680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433045.798419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433045.798422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433045.798425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433045.798448:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433045.798478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433045.798489:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433045/real 1789433045] req@ffff9811d2103b80 x1876350913687808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433045.798502:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913687808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433045.798513:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433045.798516:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433045.798519:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433045.798524:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687936:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433045.798542:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433045.798552:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433045/real 1789433045] req@ffff9811d2101500 x1876350913687936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433045.798562:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913687936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433045.798571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433045.798573:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433045.798577:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913687936:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433046.694034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433047.718109:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433048.742029:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433049.766022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433050.790000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433050.917940:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433050.917951:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433050.917956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433050.917967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5420 00000100:00080000:2.0:1789433050.917977:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433050.917989:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433050.918028:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913688064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433050.918059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433050.918063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433050.918065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433050.918070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5420 00000100:00080000:2.0:1789433050.918073:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433050.918078:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433050.918087:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433050.918108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913688192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433050.918124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433050.918127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433050.918129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433050.918132:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5420 00000100:00080000:2.0:1789433050.918135:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433050.918138:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433050.918142:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433050.918150:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913688320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433050.918157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433050.918160:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433050.918162:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433050.918165:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5420 00000100:00080000:2.0:1789433050.918168:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433050.918172:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433050.918176:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433050.918198:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913688448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433050.918268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913688064:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433050.918320:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433050.918358:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433050/real 1789433050] req@ffff9811d2102d80 x1876350913688064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433050.918371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913688064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433050.918377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433050.918380:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433050.918382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913688064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433050.918386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433050.918401:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433050.918408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433050/real 1789433050] req@ffff9811d2103800 x1876350913688192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433050.918415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913688192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433050.918431:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433050.918433:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433050.918435:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433050.918439:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433050.918454:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433050.918460:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433050/real 1789433050] req@ffff9811d2100000 x1876350913688320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433050.918468:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913688320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433050.918475:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433050.918478:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433050.918480:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433050.918485:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688448:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433050.918504:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433050.918514:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433050/real 1789433050] req@ffff9811d2101500 x1876350913688448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433050.918523:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913688448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433050.918531:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433050.918534:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433050.918537:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688448:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433051.814054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433052.838016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433053.862031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433054.886012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433055.910007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433056.037941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433056.037950:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433056.037954:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433056.037964:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5425 00000100:00080000:2.0:1789433056.037973:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433056.037983:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433056.038018:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913688576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433056.038046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433056.038050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433056.038052:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433056.038056:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5425 00000100:00080000:2.0:1789433056.038059:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433056.038070:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433056.038079:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433056.038091:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913688704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433056.038099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913688576:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433056.038104:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433056.038108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433056.038110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433056.038114:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5425 00000100:00080000:2.0:1789433056.038118:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433056.038123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433056.038127:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433056.038138:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913688832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433056.038149:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433056.038153:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433056.038155:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433056.038158:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5425 00000100:00080000:2.0:1789433056.038162:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433056.038166:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433056.038170:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433056.038176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433056.038187:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913688960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433056.038209:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433056/real 1789433056] req@ffff9811d2103b80 x1876350913688576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433056.038226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913688576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433056.038234:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433056.038238:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433056.038248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913688576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433056.038274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433056.038300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433056.038312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433056/real 1789433056] req@ffff9811d2101c00 x1876350913688704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433056.038322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913688704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433056.038330:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433056.038333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433056.038335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433056.038341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433056.038357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433056.038366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433056/real 1789433056] req@ffff9811d2102a00 x1876350913688832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433056.038375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913688832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433056.038396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433056.038399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433056.038401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433056.038407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688960:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433056.038425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433056.038436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433056/real 1789433056] req@ffff9811d2100e00 x1876350913688960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433056.038446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913688960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433056.038458:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433056.038460:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433056.038462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913688960:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433056.934002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433057.958307:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433058.982082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433060.005991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433061.030051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433061.157968:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433061.157979:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433061.157983:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433061.157993:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5430 00000100:00080000:3.0:1789433061.158001:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433061.158012:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433061.158047:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913689088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433061.158079:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433061.158083:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433061.158085:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433061.158089:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5430 00000100:00080000:3.0:1789433061.158093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433061.158098:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433061.158107:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433061.158119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913689088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433061.158124:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913689216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433061.158135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433061.158139:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433061.158141:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433061.158144:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5430 00000100:00080000:3.0:1789433061.158147:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433061.158152:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433061.158156:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433061.158167:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913689344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433061.158175:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433061.158178:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433061.158180:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433061.158183:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5430 00000100:00080000:3.0:1789433061.158187:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433061.158191:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433061.158196:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433061.158200:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433061.158221:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433061.158293:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913689472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433061.158354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689472:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433062.054037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433063.078447:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433064.102103:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433064.230400:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433064.230458:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433064.230478:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433064.230525:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433064.230534:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433064.230543:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789433064.230617:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433061/real 1789433064] req@ffff9811d1bc7100 x1876350913689472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433064.230641:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913689472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433064.230654:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433064.230659:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433064.230662:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689472:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789433064.230681:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433061/real 1789433064] req@ffff9811d1bc5500 x1876350913689216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433064.230694:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913689216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433064.230702:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433064.230706:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433064.230710:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433064.230717:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433061/real 1789433064] req@ffff9811d1bc6680 x1876350913689344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433064.230729:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913689344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433064.230737:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433064.230740:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433064.230743:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433064.230749:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433061/real 1789433064] req@ffff9811d1bc4700 x1876350913689088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433064.230759:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913689088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433064.230766:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433064.230769:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433064.230773:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913689088:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789433065.126082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433066.150125:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433066.277981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433066.277991:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433066.277995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433066.278004:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5435 00000100:00080000:3.0:1789433066.278014:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433066.278024:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433066.278060:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913689600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433066.278094:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433066.278098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433066.278100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433066.278105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5435 00000100:00080000:3.0:1789433066.278110:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433066.278118:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433066.278139:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913689600:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433066.278139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433066.278160:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913689728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433066.278172:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433066.278175:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433066.278177:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433066.278180:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5435 00000100:00080000:3.0:1789433066.278186:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433066.278192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433066.278198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789433066.278200:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433066.278214:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913689856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433066.278224:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433066.278228:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433066.278230:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433066.278235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433066/real 1789433066] req@ffff9811d1bc6300 x1876350913689600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433066.278235:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5435 00000100:00080000:3.0:1789433066.278241:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433066.278250:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433066.278260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913689600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789433066.278268:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789433066.278272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433066.278277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433066.278282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913689600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433066.278286:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913689984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433066.278317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433066.278339:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433066.278350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433066/real 1789433066] req@ffff9811d1bc5880 x1876350913689728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433066.278362:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913689728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433066.278370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433066.278373:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433066.278376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433066.278381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433066.278395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433066.278402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433066/real 1789433066] req@ffff9811d1bc5f80 x1876350913689856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433066.278413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913689856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433066.278421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433066.278423:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433066.278426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433066.278432:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689984:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433066.278446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433066.278454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433066/real 1789433066] req@ffff9811d1bc4000 x1876350913689984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433066.278465:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913689984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433066.278473:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433066.278476:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433066.278479:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913689984:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433067.174423:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433068.198082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433069.223481:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433070.246047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433071.270499:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433071.398014:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433071.398024:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433071.398030:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433071.398041:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5440 00000100:00080000:0.0:1789433071.398052:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433071.398067:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433071.398120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913690112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433071.398165:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433071.398169:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433071.398170:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433071.398174:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5440 00000100:00080000:0.0:1789433071.398178:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433071.398182:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433071.398193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433071.398204:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913690240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433071.398210:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433071.398213:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433071.398214:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433071.398216:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5440 00000100:00080000:0.0:1789433071.398219:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433071.398222:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433071.398224:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433071.398230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913690368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433071.398236:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433071.398238:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433071.398239:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433071.398242:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5440 00000100:00080000:0.0:1789433071.398243:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433071.398245:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433071.398247:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433071.398253:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913690496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433071.398313:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913690112:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789433071.398384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433071.398439:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433071/real 1789433071] req@ffff9811c8cd6a00 x1876350913690112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433071.398461:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913690112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433071.398473:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433071.398491:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433071.398495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913690112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433071.398503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433071.398528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433071.398543:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433071/real 1789433071] req@ffff9811c8cd7b80 x1876350913690240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433071.398555:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913690240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433071.398563:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433071.398566:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433071.398570:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433071.398576:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433071.398596:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433071.398608:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433071/real 1789433071] req@ffff9811c8cd6300 x1876350913690368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433071.398619:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913690368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433071.398627:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433071.398631:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433071.398634:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433071.398641:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690496:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433071.398664:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433071.398675:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433071/real 1789433071] req@ffff9811c8cd4000 x1876350913690496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433071.398686:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913690496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433071.398694:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433071.398697:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433071.398698:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433072.294036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433073.319141:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433074.342051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433075.365976:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433076.390057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433076.517941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433076.517951:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433076.517955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433076.517965:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5446 00000100:00080000:0.0:1789433076.517974:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433076.517984:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433076.518020:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913690624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433076.518051:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433076.518055:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433076.518057:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433076.518065:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5446 00000100:00080000:0.0:1789433076.518068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433076.518073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433076.518082:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433076.518095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913690752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433076.518104:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433076.518107:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433076.518110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433076.518113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5446 00000100:00080000:0.0:1789433076.518116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433076.518121:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433076.518126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433076.518137:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913690880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433076.518147:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433076.518150:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433076.518153:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433076.518157:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5446 00000100:00080000:0.0:1789433076.518160:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433076.518165:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433076.518170:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433076.518171:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913690624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433076.518185:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913691008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789433076.518228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433076.518281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433076/real 1789433076] req@ffff9811c8cd5180 x1876350913690624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433076.518293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913690624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433076.518304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433076.518307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433076.518309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913690624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433076.518316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433076.518337:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433076.518349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433076/real 1789433076] req@ffff9811c8cd7800 x1876350913690752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433076.518361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913690752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433076.518368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433076.518371:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433076.518374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433076.518380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433076.518399:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433076.518416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433076/real 1789433076] req@ffff9811c8cd7480 x1876350913690880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433076.518424:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913690880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433076.518429:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433076.518431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433076.518433:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913690880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433076.518453:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691008:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433076.518473:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433076.518493:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433076/real 1789433076] req@ffff9811c8cd7100 x1876350913691008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433076.518504:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913691008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433076.518513:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433076.518516:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433076.518519:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433077.414290:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433078.438796:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433079.462142:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433080.486006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433081.509960:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433081.637955:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433081.637965:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433081.637970:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433081.637979:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5451 00000100:00080000:1.0:1789433081.637986:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433081.637997:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433081.638027:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913691136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433081.638049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433081.638058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433081.638060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433081.638064:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5451 00000100:00080000:1.0:1789433081.638067:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433081.638072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433081.638082:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433081.638094:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913691264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433081.638103:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433081.638106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433081.638107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433081.638109:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5451 00000100:00080000:1.0:1789433081.638112:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433081.638115:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433081.638118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433081.638125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913691392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433081.638131:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433081.638133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433081.638134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433081.638136:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5451 00000100:00080000:1.0:1789433081.638137:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433081.638139:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433081.638141:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433081.638146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913691520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433081.638214:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913691136:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433081.638286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433081.638339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433081/real 1789433081] req@ffff9811d0bde680 x1876350913691136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433081.638358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913691136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433081.638368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433081.638372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433081.638376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913691136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433081.638384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433081.638417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433081.638430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433081/real 1789433081] req@ffff9811d0bdf480 x1876350913691264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433081.638441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913691264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433081.638449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433081.638452:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433081.638455:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433081.638460:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433081.638478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433081.638489:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433081/real 1789433081] req@ffff9811d0bdfb80 x1876350913691392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433081.638504:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913691392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433081.638511:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433081.638515:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433081.638518:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433081.638523:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691520:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433081.638542:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433081.638554:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433081/real 1789433081] req@ffff9811d0bdc700 x1876350913691520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433081.638565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913691520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433081.638573:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433081.638576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433081.638579:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433082.534016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433083.557987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433084.582063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433085.605993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433086.630041:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433086.757909:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433086.757918:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433086.757921:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433086.757930:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5456 00000100:00080000:1.0:1789433086.757938:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433086.757947:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433086.757976:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913691648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433086.757997:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433086.758000:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433086.758002:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433086.758010:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5456 00000100:00080000:1.0:1789433086.758013:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433086.758018:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433086.758027:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433086.758041:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913691776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433086.758045:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913691648:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433086.758061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433086.758064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433086.758065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433086.758070:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5456 00000100:00080000:1.0:1789433086.758074:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433086.758079:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433086.758084:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433086.758096:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913691904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433086.758107:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433086.758111:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433086.758113:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433086.758117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433086.758122:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5456 00000100:00080000:1.0:1789433086.758126:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433086.758131:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433086.758136:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433086.758149:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913692032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433086.758170:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433086/real 1789433086] req@ffff9811d0bdea00 x1876350913691648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433086.758191:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913691648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433086.758202:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433086.758206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433086.758210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913691648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433086.758232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433086.758254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433086.758273:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433086/real 1789433086] req@ffff9811d0bded80 x1876350913691776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433086.758296:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913691776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433086.758304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433086.758307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433086.758310:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433086.758316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433086.758335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433086.758344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433086/real 1789433086] req@ffff9811d0bdc000 x1876350913691904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433086.758351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913691904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433086.758356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433086.758358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433086.758360:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913691904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433086.758363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692032:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433086.758374:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433086.758379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433086/real 1789433086] req@ffff9811d0bdd880 x1876350913692032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433086.758386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913692032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433086.758395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433086.758397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433086.758399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433087.654147:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433088.678107:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433089.702056:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433090.726111:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433091.750137:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433091.877954:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433091.877964:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433091.877969:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433091.877978:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5461 00000100:00080000:1.0:1789433091.877989:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433091.878000:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433091.878043:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913692160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433091.878070:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433091.878072:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433091.878074:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433091.878077:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5461 00000100:00080000:1.0:1789433091.878079:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433091.878082:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433091.878092:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433091.878100:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913692288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433091.878109:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433091.878113:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433091.878115:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433091.878119:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5461 00000100:00080000:1.0:1789433091.878132:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789433091.878136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913692160:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433091.878139:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433091.878148:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433091.878165:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913692416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433091.878177:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433091.878181:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433091.878183:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433091.878186:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5461 00000100:00080000:1.0:1789433091.878199:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433091.878204:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433091.878210:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433091.878221:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433091.878225:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913692544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433091.878281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433091/real 1789433091] req@ffff9811d0bdf100 x1876350913692160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433091.878298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913692160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433091.878309:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433091.878313:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433091.878317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913692160:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433091.878325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433091.878348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433091.878361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433091/real 1789433091] req@ffff9811d0bde300 x1876350913692288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433091.878373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913692288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433091.878380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433091.878383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433091.878385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433091.878407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433091.878424:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433091.878434:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433091/real 1789433091] req@ffff9811d0bdf800 x1876350913692416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433091.878446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913692416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433091.878454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433091.878458:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433091.878461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433091.878467:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692544:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433091.878490:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433091.878499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433091/real 1789433091] req@ffff9811d0bdd880 x1876350913692544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433091.878507:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913692544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433091.878513:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433091.878515:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433091.878517:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433092.774069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433093.798131:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433094.822112:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433095.846012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433096.870053:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433096.997974:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433096.997986:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433096.997991:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433096.998005:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5466 00000100:00080000:2.0:1789433096.998016:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433096.998030:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433096.998076:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913692672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433096.998122:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433096.998125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433096.998127:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433096.998131:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5466 00000100:00080000:2.0:1789433096.998134:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433096.998139:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433096.998150:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433096.998161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913692800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433096.998169:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433096.998172:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433096.998173:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433096.998176:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5466 00000100:00080000:2.0:1789433096.998179:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433096.998182:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433096.998186:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433096.998194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913692928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433096.998203:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433096.998207:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433096.998208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433096.998213:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5466 00000100:00080000:2.0:1789433096.998215:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433096.998220:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433096.998225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913692672:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789433096.998226:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433096.998239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913693056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433096.998317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433096.998375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433096/real 1789433096] req@ffff9811d2101f80 x1876350913692672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433096.998394:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913692672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433096.998406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433096.998410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433096.998414:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913692672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433096.998421:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433096.998444:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433096.998457:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433096/real 1789433096] req@ffff9811d2101180 x1876350913692800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433096.998468:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913692800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433096.998475:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433096.998479:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433096.998482:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433096.998488:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433096.998506:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433096.998515:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433096/real 1789433096] req@ffff9811d2101880 x1876350913692928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433096.998534:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913692928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433096.998541:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433096.998544:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433096.998547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913692928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433096.998574:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433096.998596:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433096.998607:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433096/real 1789433096] req@ffff9811d2100e00 x1876350913693056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433096.998617:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913693056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433096.998625:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433096.998627:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433096.998630:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433097.894093:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433098.919126:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433099.942099:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433100.967108:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433101.990009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433102.117937:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433102.117945:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433102.117948:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433102.117956:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5471 00000100:00080000:3.0:1789433102.117963:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433102.117972:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433102.118000:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913693184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433102.118019:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433102.118021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433102.118022:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433102.118029:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5471 00000100:00080000:3.0:1789433102.118031:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433102.118034:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433102.118041:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433102.118062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913693312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433102.118073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913693184:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433102.118075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433102.118087:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433102.118089:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433102.118093:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5471 00000100:00080000:3.0:1789433102.118097:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433102.118102:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433102.118107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433102.118117:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913693440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433102.118125:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433102.118127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433102.118129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433102.118132:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5471 00000100:00080000:3.0:1789433102.118136:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433102.118137:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433102.118143:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433102.118149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433102.118163:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913693568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433102.118185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433102/real 1789433102] req@ffff9811d1bc5180 x1876350913693184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433102.118200:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913693184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433102.118208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433102.118211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433102.118215:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913693184:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433102.118237:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433102.118259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433102.118269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433102/real 1789433102] req@ffff9811d1bc5c00 x1876350913693312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433102.118277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913693312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433102.118282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433102.118284:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433102.118286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433102.118289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433102.118300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433102.118305:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433102/real 1789433102] req@ffff9811d1bc7800 x1876350913693440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433102.118312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913693440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433102.118316:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433102.118318:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433102.118319:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433102.118323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693568:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433102.118332:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433102.118337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433102/real 1789433102] req@ffff9811d1bc7480 x1876350913693568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433102.118343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913693568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433102.118348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433102.118350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433102.118351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433103.013973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433104.038009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433105.062023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433106.086013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433107.110006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433107.237913:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433107.237922:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433107.237925:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433107.237935:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5476 00000100:00080000:0.0:1789433107.237943:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433107.237953:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433107.237994:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913693696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433107.238027:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433107.238031:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433107.238033:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433107.238037:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5476 00000100:00080000:0.0:1789433107.238041:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433107.238046:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433107.238060:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433107.238062:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913693696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433107.238076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913693824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433107.238084:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433107.238086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433107.238088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433107.238090:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5476 00000100:00080000:0.0:1789433107.238093:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433107.238096:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433107.238100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433107.238108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913693952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433107.238115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433107.238119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1789433107.238119:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433107.238123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433107.238126:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5476 00000100:00080000:0.0:1789433107.238128:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433107.238131:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433107.238134:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433107.238141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913694080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433107.238160:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433107/real 1789433107] req@ffff9811c8cd6680 x1876350913693696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433107.238178:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913693696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433107.238187:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433107.238191:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433107.238195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913693696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433107.238215:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433107.238234:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433107.238243:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433107/real 1789433107] req@ffff9811c8cd4700 x1876350913693824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433107.238255:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913693824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433107.238260:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433107.238262:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433107.238264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433107.238269:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433107.238285:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433107.238296:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433107/real 1789433107] req@ffff9811c8cd5c00 x1876350913693952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433107.238313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913693952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433107.238327:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433107.238329:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433107.238331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913693952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433107.238335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433107.238349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433107.238356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433107/real 1789433107] req@ffff9811c8cd7100 x1876350913694080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433107.238363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913694080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433107.238367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433107.238369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433107.238371:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433108.134014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433109.158012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433110.182003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433111.205970:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433112.230001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433112.357931:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433112.357941:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433112.357945:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433112.357954:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5481 00000100:00080000:0.0:1789433112.357962:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433112.357983:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433112.358015:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913694208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433112.358040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433112.358043:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433112.358045:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433112.358050:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5481 00000100:00080000:0.0:1789433112.358053:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433112.358058:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433112.358067:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433112.358078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913694336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433112.358079:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913694208:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433112.358088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433112.358090:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433112.358092:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433112.358095:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5481 00000100:00080000:0.0:1789433112.358098:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433112.358102:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433112.358106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433112.358114:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913694464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433112.358123:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433112.358125:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433112.358126:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433112.358129:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5481 00000100:00080000:0.0:1789433112.358132:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789433112.358132:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433112.358136:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433112.358139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433112.358148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913694592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433112.358166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433112/real 1789433112] req@ffff9811c8cd7480 x1876350913694208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433112.358182:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913694208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433112.358192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433112.358197:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433112.358200:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913694208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433112.358222:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433112.358243:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433112.358253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433112/real 1789433112] req@ffff9811c8cd7800 x1876350913694336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433112.358260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913694336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433112.358265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433112.358272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433112.358274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433112.358278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433112.358289:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433112.358295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433112/real 1789433112] req@ffff9811c8cd5180 x1876350913694464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433112.358301:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913694464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433112.358308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433112.358311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433112.358314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433112.358319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694592:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433112.358337:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433112.358348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433112/real 1789433112] req@ffff9811c8cd4000 x1876350913694592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433112.358359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913694592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433112.358367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433112.358370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433112.358373:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433113.254012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433114.278905:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433115.301986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433116.325974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433117.350000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433117.478107:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433117.478117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433117.478120:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433117.478130:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5487 00000100:00080000:1.0:1789433117.478138:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433117.478149:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433117.478181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913694720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433117.478206:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433117.478212:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433117.478215:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433117.478218:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5487 00000100:00080000:1.0:1789433117.478222:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433117.478227:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433117.478236:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433117.478249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913694720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433117.478250:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913694848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433117.478269:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433117.478271:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433117.478273:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433117.478276:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5487 00000100:00080000:1.0:1789433117.478279:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433117.478282:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433117.478285:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433117.478294:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913694976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433117.478300:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433117.478302:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433117.478303:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433117.478305:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5487 00000100:00080000:1.0:1789433117.478307:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433117.478312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433117.478312:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433117.478319:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433117.478326:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913695104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433117.478371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433117/real 1789433117] req@ffff9811d0bdc000 x1876350913694720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433117.478389:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913694720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433117.478400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433117.478405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433117.478409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913694720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433117.478440:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433117.478463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433117.478475:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433117/real 1789433117] req@ffff9811d0bded80 x1876350913694848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433117.478487:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913694848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433117.478495:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433117.478498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433117.478502:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433117.478507:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433117.478526:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433117.478541:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433117/real 1789433117] req@ffff9811d0bdea00 x1876350913694976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433117.478549:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913694976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433117.478554:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433117.478556:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433117.478558:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913694976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433117.478562:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433117.478573:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433117.478579:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433117/real 1789433117] req@ffff9811d0bdc700 x1876350913695104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433117.478589:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913695104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433117.478596:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433117.478608:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433117.478611:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433118.373974:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433119.397989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433120.421981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433121.445999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433122.469957:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433122.597948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433122.597955:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433122.597959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433122.597966:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5492 00000100:00080000:0.0:1789433122.597973:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433122.597981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433122.598009:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913695232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433122.598028:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433122.598031:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433122.598033:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433122.598038:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5492 00000100:00080000:0.0:1789433122.598041:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433122.598046:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433122.598055:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433122.598062:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913695232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433122.598070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913695360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433122.598088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433122.598091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433122.598094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433122.598098:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5492 00000100:00080000:0.0:1789433122.598104:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433122.598109:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433122.598114:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789433122.598118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433122.598128:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913695488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433122.598139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433122.598141:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433122.598143:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433122.598146:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5492 00000100:00080000:0.0:1789433122.598149:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433122.598156:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433122.598158:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433122/real 1789433122] req@ffff9811c8cd6300 x1876350913695232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789433122.598162:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433122.598179:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913695616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433122.598188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913695232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433122.598204:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433122.598208:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433122.598212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913695232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433122.598229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433122.598250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433122.598262:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433122/real 1789433122] req@ffff9811c8cd7b80 x1876350913695360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433122.598274:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913695360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433122.598282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433122.598285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433122.598289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433122.598294:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433122.598314:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433122.598325:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433122/real 1789433122] req@ffff9811c8cd6a00 x1876350913695488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433122.598337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913695488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433122.598350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433122.598354:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433122.598357:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433122.598362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433122.598380:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433122.598389:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433122/real 1789433122] req@ffff9811c8cd4a80 x1876350913695616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433122.598399:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913695616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433122.598407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433122.598410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433122.598413:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433123.494006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433124.518012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433125.542027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433126.566070:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433127.590033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433127.717944:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433127.717954:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433127.717958:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433127.717967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5497 00000100:00080000:3.0:1789433127.717975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433127.717986:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433127.718020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913695744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433127.718070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789433127.718073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913695744:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433127.718075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433127.718081:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433127.718086:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5497 00000100:00080000:3.0:1789433127.718091:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433127.718097:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433127.718106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433127.718120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913695872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433127.718131:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433127.718135:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433127.718136:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433127.718140:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5497 00000100:00080000:3.0:1789433127.718144:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433127.718150:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433127.718155:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695872:192.168.203.128@tcp:38:kworker.0 00000008:00100000:3.0:1789433127.718156:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433127.718172:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913696000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433127.718190:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433127.718193:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433127.718195:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433127.718200:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5497 00000100:00080000:3.0:1789433127.718205:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789433127.718210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696000:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789433127.718213:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433127.718221:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433127.718232:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913696128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433127.718251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433128.614018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433129.638283:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433130.661998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789433130.790178:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789433130.790220:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789433130.790234:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433130.790267:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433130.790279:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433130.790289:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433130.790297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433127/real 1789433130] req@ffff9811d1bc7480 x1876350913695872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433130.790322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913695872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433130.790339:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433130.790344:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433130.790348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913695872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433130.790355:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433127/real 1789433130] req@ffff9811d1bc6a00 x1876350913695744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433130.790366:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913695744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433130.790374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433130.790377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433130.790379:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913695744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433130.790399:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433127/real 1789433130] req@ffff9811d1bc5c00 x1876350913696128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433130.790410:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913696128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433130.790418:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433130.790422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433130.790425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696128:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433130.790431:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433127/real 1789433130] req@ffff9811d1bc7800 x1876350913696000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433130.790442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913696000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433130.790449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433130.790453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433130.790456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696000:192.168.203.128@tcp:38:kworker.0 00010000:00010000:3.0:1789433131.685995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433132.710071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433132.837934:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433132.837944:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433132.837948:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433132.837957:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5502 00000100:00080000:3.0:1789433132.837966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433132.837976:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433132.838019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913696256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433132.838044:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433132.838048:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433132.838050:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433132.838054:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5502 00000100:00080000:3.0:1789433132.838057:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789433132.838084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913696256:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433132.838107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433132.838117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433132.838132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913696384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433132.838149:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433132.838153:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433132.838156:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433132.838157:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433132.838165:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5502 00000100:00080000:3.0:1789433132.838169:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433132.838175:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433132.838180:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433132.838193:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913696512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433132.838200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433132/real 1789433132] req@ffff9811d1bc5180 x1876350913696256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433132.838207:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433132.838216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433132.838219:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433132.838224:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5502 00000100:00100000:0.0:1789433132.838228:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913696256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433132.838229:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433132.838238:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00080000:0.0:1789433132.838244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789433132.838245:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789433132.838252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433132.838258:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913696256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433132.838261:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913696640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433132.838285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433132.838373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433132.838387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433132/real 1789433132] req@ffff9811d1bc4000 x1876350913696384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433132.838398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913696384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433132.838406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433132.838410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433132.838414:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433132.838420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433132.838439:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433132.838450:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433132/real 1789433132] req@ffff9811d1bc5f80 x1876350913696512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433132.838461:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913696512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433132.838469:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433132.838472:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433132.838475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433132.838481:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696640:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433132.838500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433132.838510:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433132/real 1789433132] req@ffff9811d1bc5880 x1876350913696640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433132.838521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913696640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433132.838530:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433132.838533:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433132.838536:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433133.733998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433134.758016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433135.782017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433136.806016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433137.831127:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433137.958042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433137.958051:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433137.958054:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433137.958062:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5507 00000100:00080000:0.0:1789433137.958068:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433137.958077:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433137.958110:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913696768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433137.958129:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433137.958132:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433137.958137:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433137.958141:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5507 00000100:00080000:0.0:1789433137.958144:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433137.958148:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433137.958157:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433137.958164:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913696768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433137.958168:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913696896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433137.958176:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433137.958178:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433137.958180:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433137.958183:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5507 00000100:00080000:0.0:1789433137.958185:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433137.958189:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433137.958192:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433137.958200:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913697024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433137.958213:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789433137.958215:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433137.958216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433137.958217:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433137.958220:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5507 00000100:00080000:0.0:1789433137.958223:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433137.958227:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433137.958230:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433137.958239:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913697152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433137.958255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433137/real 1789433137] req@ffff9811c8cd5500 x1876350913696768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433137.958272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913696768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433137.958282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433137.958285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433137.958289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913696768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433137.958309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433137.958331:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433137.958343:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433137/real 1789433137] req@ffff9811c8cd6d80 x1876350913696896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433137.958355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913696896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433137.958360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433137.958362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433137.958364:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913696896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433137.958368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433137.958381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433137.958387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433137/real 1789433137] req@ffff9811c8cd5f80 x1876350913697024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433137.958394:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913697024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433137.958398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433137.958399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433137.958401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433137.958404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697152:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433137.958418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433137.958427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433137/real 1789433137] req@ffff9811c8cd4a80 x1876350913697152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433137.958437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913697152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433137.958445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433137.958447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433137.958450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433138.854021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433139.878027:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433140.902044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433141.925980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433142.950065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433143.077986:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433143.077995:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433143.077999:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433143.078008:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5512 00000100:00080000:0.0:1789433143.078017:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433143.078028:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433143.078062:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913697280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433143.078086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433143.078089:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433143.078091:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433143.078095:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5512 00000100:00080000:0.0:1789433143.078098:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433143.078103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433143.078113:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433143.078126:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913697408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433143.078138:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433143.078144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433143.078151:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789433143.078151:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913697280:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433143.078162:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5512 00000100:00080000:0.0:1789433143.078168:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433143.078173:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433143.078178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433143.078194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913697536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433143.078206:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433143.078210:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433143.078212:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433143.078216:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5512 00000100:00080000:0.0:1789433143.078221:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433143.078227:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433143.078233:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789433143.078237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433143.078257:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913697664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433143.078301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433143/real 1789433143] req@ffff9811c8cd6a00 x1876350913697280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433143.078332:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913697280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433143.078346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433143.078349:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433143.078352:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913697280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433143.078377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433143.078396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433143.078406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433143/real 1789433143] req@ffff9811c8cd7b80 x1876350913697408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433143.078414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913697408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433143.078419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433143.078421:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433143.078423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433143.078427:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433143.078436:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433143.078441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433143/real 1789433143] req@ffff9811c8cd6300 x1876350913697536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433143.078448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913697536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433143.078457:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433143.078459:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433143.078461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433143.078464:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433143.078474:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433143.078480:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433143/real 1789433143] req@ffff9811c8cd4000 x1876350913697664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433143.078485:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913697664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433143.078490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433143.078493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433143.078494:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433143.974115:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433144.998024:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433146.022030:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433147.046004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433148.070025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433148.197938:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433148.197946:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433148.197949:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433148.197961:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5517 00000100:00080000:2.0:1789433148.197968:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433148.197975:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433148.198004:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913697792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433148.198024:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433148.198031:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433148.198033:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433148.198037:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5517 00000100:00080000:2.0:1789433148.198041:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433148.198046:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433148.198055:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433148.198066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913697792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433148.198070:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913697920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433148.198082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433148.198086:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433148.198088:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433148.198091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5517 00000100:00080000:2.0:1789433148.198094:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433148.198099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433148.198103:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433148.198112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913698048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433148.198121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433148.198125:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433148.198128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789433148.198129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433148.198136:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5517 00000100:00080000:2.0:1789433148.198141:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433148.198147:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433148.198152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433148.198165:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913698176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433148.198186:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433148/real 1789433148] req@ffff9811d2102a00 x1876350913697792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433148.198205:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913697792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433148.198215:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433148.198219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433148.198222:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913697792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433148.198243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433148.198264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433148.198276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433148/real 1789433148] req@ffff9811d2101c00 x1876350913697920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433148.198287:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913697920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433148.198294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433148.198297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433148.198300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913697920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433148.198305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433148.198321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433148.198330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433148/real 1789433148] req@ffff9811d2103b80 x1876350913698048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433148.198348:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913698048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433148.198357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433148.198360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433148.198363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433148.198369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433148.198387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433148.198397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433148/real 1789433148] req@ffff9811d2101500 x1876350913698176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433148.198418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913698176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433148.198425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433148.198428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433148.198431:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433149.094052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433150.117994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433151.142092:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433152.166067:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433153.190122:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433153.317989:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433153.318003:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433153.318008:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433153.318022:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5522 00000100:00080000:0.0:1789433153.318033:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433153.318046:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433153.318093:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913698304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433153.318123:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433153.318127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433153.318129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433153.318133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5522 00000100:00080000:0.0:1789433153.318136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433153.318142:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433153.318154:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433153.318167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913698304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433153.318169:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913698432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433153.318190:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433153.318194:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433153.318196:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433153.318201:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5522 00000100:00080000:0.0:1789433153.318205:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433153.318210:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433153.318215:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433153.318225:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913698560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433153.318240:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433153.318243:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433153.318245:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789433153.318250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433153.318251:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5522 00000100:00080000:0.0:1789433153.318257:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433153.318262:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433153.318266:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433153.318280:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913698688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433153.318337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433153/real 1789433153] req@ffff9811c8cd5180 x1876350913698304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433153.318359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913698304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433153.318371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433153.318375:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433153.318379:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913698304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433153.318405:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433153.318430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433153.318442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433153/real 1789433153] req@ffff9811c8cd7800 x1876350913698432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433153.318462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913698432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433153.318477:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433153.318480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433153.318484:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433153.318490:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433153.318508:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433153.318519:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433153/real 1789433153] req@ffff9811c8cd7480 x1876350913698560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433153.318529:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913698560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433153.318537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433153.318540:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433153.318543:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433153.318549:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698688:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433153.318577:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433153.318587:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433153/real 1789433153] req@ffff9811c8cd7100 x1876350913698688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433153.318598:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913698688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433153.318606:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433153.318609:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433153.318613:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433154.214102:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433155.238096:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433156.262032:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433157.285991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433158.310039:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433158.438006:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433158.438013:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433158.438016:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433158.438024:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5528 00000100:00080000:2.0:1789433158.438030:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433158.438040:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433158.438068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913698816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433158.438088:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433158.438092:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433158.438093:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433158.438097:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5528 00000100:00080000:2.0:1789433158.438101:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433158.438106:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433158.438115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433158.438123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913698816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433158.438130:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913698944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433158.438140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433158.438142:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433158.438143:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433158.438146:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5528 00000100:00080000:2.0:1789433158.438148:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433158.438152:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433158.438155:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433158.438164:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913699072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433158.438170:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433158.438173:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433158.438174:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433158.438176:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5528 00000100:00080000:2.0:1789433158.438184:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433158.438187:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433158.438190:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789433158.438193:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433158.438207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913699200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433158.438233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433158/real 1789433158] req@ffff9811d2100000 x1876350913698816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433158.438251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913698816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433158.438262:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433158.438266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433158.438270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913698816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433158.438291:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433158.438313:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433158.438324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433158/real 1789433158] req@ffff9811d2103800 x1876350913698944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433158.438335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913698944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433158.438343:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433158.438346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433158.438349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913698944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433158.438354:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433158.438369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433158.438379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433158/real 1789433158] req@ffff9811d2102d80 x1876350913699072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433158.438390:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913699072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433158.438397:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433158.438399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433158.438403:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433158.438408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433158.438423:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433158.438440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433158/real 1789433158] req@ffff9811d2101500 x1876350913699200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433158.438451:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913699200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433158.438459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433158.438462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433158.438465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433159.334057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433160.358082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433161.382052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433162.405997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433163.429989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433163.557983:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433163.557993:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433163.557997:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433163.558007:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5533 00000100:00080000:2.0:1789433163.558015:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433163.558026:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433163.558057:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913699328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433163.558085:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433163.558089:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433163.558090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433163.558094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5533 00000100:00080000:2.0:1789433163.558097:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433163.558101:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433163.558110:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433163.558121:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913699456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433163.558130:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433163.558133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433163.558134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433163.558138:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5533 00000100:00080000:2.0:1789433163.558141:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433163.558144:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433163.558148:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433163.558156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913699584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433163.558165:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433163.558167:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433163.558169:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433163.558172:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5533 00000100:00080000:2.0:1789433163.558175:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433163.558178:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433163.558182:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433163.558190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913699712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789433163.558323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913699328:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789433163.558377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433163.558411:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433163/real 1789433163] req@ffff9811d2103b80 x1876350913699328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433163.558426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913699328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433163.558437:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433163.558441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433163.558445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913699328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433163.558451:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433163.558472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433163.558484:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433163/real 1789433163] req@ffff9811d2101c00 x1876350913699456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433163.558495:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913699456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433163.558503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433163.558506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433163.558508:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433163.558513:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433163.558531:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433163.558541:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433163/real 1789433163] req@ffff9811d2102a00 x1876350913699584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433163.558551:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913699584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433163.558559:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433163.558562:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433163.558565:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433163.558571:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433163.558588:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433163.558611:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433163/real 1789433163] req@ffff9811d2100e00 x1876350913699712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433163.558622:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913699712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433163.558630:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433163.558632:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433163.558635:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433164.454003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433165.477990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433166.502003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433167.526095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433168.550043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433168.677943:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433168.677962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433168.677966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433168.677976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5538 00000100:00080000:0.0:1789433168.677983:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433168.678000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433168.678034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913699840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433168.678055:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433168.678058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433168.678059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433168.678065:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5538 00000100:00080000:0.0:1789433168.678069:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433168.678073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433168.678081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433168.678088:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913699840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433168.678092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913699968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433168.678102:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433168.678104:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433168.678105:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433168.678107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5538 00000100:00080000:0.0:1789433168.678108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433168.678111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433168.678113:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433168.678119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913700096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433168.678126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433168.678128:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433168.678129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433168.678131:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5538 00000100:00080000:0.0:1789433168.678139:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433168.678142:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433168.678145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433168.678153:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913700224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789433168.678155:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433168.678194:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433168/real 1789433168] req@ffff9811c8cd5c00 x1876350913699840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433168.678212:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913699840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433168.678222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433168.678225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433168.678229:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913699840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433168.678250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433168.678272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433168.678281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433168/real 1789433168] req@ffff9811c8cd4700 x1876350913699968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433168.678293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913699968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433168.678301:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433168.678304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433168.678307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913699968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433168.678320:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789433168.678335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433168.678345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433168/real 1789433168] req@ffff9811c8cd6680 x1876350913700096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433168.678352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913700096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433168.678357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433168.678360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433168.678363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789433168.678369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700224:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789433168.678388:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433168.678398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433168/real 1789433168] req@ffff9811c8cd7100 x1876350913700224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433168.678415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913700224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433168.678432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789433168.678434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789433168.678438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433169.573962:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433170.597999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433171.622028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433172.646116:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433173.670010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433173.797974:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433173.797984:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433173.797989:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433173.797999:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5543 00000100:00080000:0.0:1789433173.798009:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433173.798019:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433173.798053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913700352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433173.798077:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433173.798081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433173.798083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433173.798087:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5543 00000100:00080000:0.0:1789433173.798090:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433173.798095:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433173.798106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433173.798118:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913700480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433173.798127:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433173.798129:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433173.798131:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433173.798133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5543 00000100:00080000:0.0:1789433173.798136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433173.798141:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433173.798144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433173.798154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913700608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433173.798167:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433173.798170:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433173.798172:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433173.798174:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5543 00000100:00080000:0.0:1789433173.798177:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433173.798180:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433173.798184:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433173.798192:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913700736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433173.798340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913700352:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433173.798398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433173.798435:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433173/real 1789433173] req@ffff9811c8cd7480 x1876350913700352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433173.798454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913700352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433173.798462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433173.798464:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433173.798467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913700352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433173.798472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433173.798491:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433173.798503:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433173/real 1789433173] req@ffff9811c8cd7800 x1876350913700480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433173.798514:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913700480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433173.798534:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433173.798538:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433173.798541:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433173.798547:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433173.798563:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433173.798576:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433173/real 1789433173] req@ffff9811c8cd5180 x1876350913700608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433173.798583:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913700608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433173.798588:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433173.798590:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433173.798592:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433173.798596:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433173.798606:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433173.798611:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433173/real 1789433173] req@ffff9811c8cd4000 x1876350913700736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433173.798618:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913700736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433173.798622:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433173.798624:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433173.798626:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433174.693990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433175.718020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433176.742061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433177.766007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433178.789985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433178.917983:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433178.917993:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433178.917997:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433178.918018:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5548 00000100:00080000:1.0:1789433178.918026:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433178.918038:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433178.918074:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913700864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433178.918106:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433178.918110:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433178.918112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433178.918116:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5548 00000100:00080000:1.0:1789433178.918119:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433178.918124:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433178.918143:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433178.918156:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913700992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433178.918167:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433178.918170:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433178.918172:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433178.918177:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5548 00000100:00080000:1.0:1789433178.918181:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433178.918185:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433178.918191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433178.918197:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913700864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433178.918208:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913701120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433178.918222:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433178.918226:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433178.918228:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433178.918234:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5548 00000100:00080000:1.0:1789433178.918238:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433178.918243:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433178.918249:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433178.918262:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913701248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433178.918306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433178.918371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433178/real 1789433178] req@ffff9811d0bdfb80 x1876350913700864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433178.918392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913700864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433178.918401:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433178.918404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433178.918407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913700864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433178.918413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433178.918431:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433178.918442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433178/real 1789433178] req@ffff9811d0bdf480 x1876350913700992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433178.918454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913700992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433178.918462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433178.918465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433178.918468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913700992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433178.918511:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433178.918532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433178.918544:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433178/real 1789433178] req@ffff9811d0bde680 x1876350913701120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433178.918560:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913701120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433178.918568:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433178.918571:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433178.918574:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433178.918579:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701248:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433178.918600:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433178.918611:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433178/real 1789433178] req@ffff9811d0bdd500 x1876350913701248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433178.918619:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913701248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433178.918625:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433178.918627:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433178.918629:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433179.814068:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433180.838016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433181.861991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433182.885986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433183.910018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433184.037914:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433184.037925:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433184.037929:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433184.037939:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5553 00000100:00080000:3.0:1789433184.037947:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433184.037958:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433184.037990:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913701376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433184.038014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433184.038016:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433184.038017:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433184.038020:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5553 00000100:00080000:3.0:1789433184.038023:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433184.038027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433184.038035:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433184.038045:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913701504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433184.038049:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913701376:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433184.038063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433184.038067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433184.038069:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433184.038074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5553 00000100:00080000:3.0:1789433184.038078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433184.038083:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433184.038088:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433184.038099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913701632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433184.038106:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433184.038109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433184.038112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433184.038113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433184.038117:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5553 00000100:00080000:3.0:1789433184.038120:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433184.038126:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433184.038129:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433184.038138:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913701760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433184.038143:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433184/real 1789433184] req@ffff9811d1bc6300 x1876350913701376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433184.038159:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913701376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433184.038167:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433184.038170:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433184.038172:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913701376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433184.038188:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433184.038202:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433184.038209:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433184/real 1789433184] req@ffff9811d1bc4700 x1876350913701504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433184.038217:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913701504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433184.038222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433184.038223:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433184.038225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433184.038228:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433184.038237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433184.038241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433184/real 1789433184] req@ffff9811d1bc6680 x1876350913701632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433184.038248:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913701632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433184.038254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433184.038257:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433184.038260:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433184.038274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433184.038294:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433184.038305:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433184/real 1789433184] req@ffff9811d1bc5500 x1876350913701760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433184.038316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913701760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433184.038329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433184.038333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433184.038336:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913701760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433184.933978:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433185.957995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433186.982517:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433188.005980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433189.032056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433189.157920:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433189.157931:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433189.157935:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433189.157945:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5558 00000100:00080000:3.0:1789433189.157954:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433189.157964:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433189.157999:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913701888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433189.158025:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433189.158028:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433189.158030:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433189.158035:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5558 00000100:00080000:3.0:1789433189.158039:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433189.158044:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433189.158053:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433189.158059:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913701888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433189.158068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913702016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433189.158084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433189.158087:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433189.158089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433189.158092:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5558 00000100:00080000:3.0:1789433189.158095:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433189.158100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433189.158104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433189.158116:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913702144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433189.158126:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433189.158131:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433189.158134:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433189.158137:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433189.158141:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5558 00000100:00080000:3.0:1789433189.158146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433189.158151:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433189.158157:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433189.158165:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433189/real 1789433189] req@ffff9811d1bc7100 x1876350913701888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433189.158171:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913702272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433189.158182:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913701888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433189.158190:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433189.158192:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433189.158196:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913701888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433189.158213:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433189.158227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433189.158235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433189/real 1789433189] req@ffff9811d1bc5500 x1876350913702016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433189.158242:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913702016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433189.158246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433189.158248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433189.158250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433189.158254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433189.158263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433189.158283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433189/real 1789433189] req@ffff9811d1bc6680 x1876350913702144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433189.158295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913702144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433189.158302:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433189.158305:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433189.158308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433189.158313:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433189.158330:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433189.158340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433189/real 1789433189] req@ffff9811d1bc4700 x1876350913702272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433189.158351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913702272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433189.158357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433189.158360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433189.158362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433190.053988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433191.077987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433192.102017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433193.127870:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433194.150008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433194.277948:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433194.277958:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433194.277962:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433194.277977:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5563 00000100:00080000:2.0:1789433194.277985:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433194.278005:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433194.278040:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913702400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433194.278075:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433194.278079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433194.278080:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433194.278085:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5563 00000100:00080000:2.0:1789433194.278088:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433194.278093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433194.278101:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433194.278112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913702528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433194.278120:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433194.278123:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433194.278124:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433194.278128:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5563 00000100:00080000:2.0:1789433194.278130:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433194.278134:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433194.278138:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433194.278145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913702656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433194.278153:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433194.278156:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433194.278157:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433194.278160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5563 00000100:00080000:2.0:1789433194.278163:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433194.278166:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433194.278170:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433194.278179:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913702784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433194.278426:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913702400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433194.278518:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433194.278550:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433194.278572:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433195.174050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433196.198180:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433197.222040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789433197.350053:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789433197.350087:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789433197.350099:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433197.350130:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433197.350139:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433197.350148:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433197.350177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433194/real 1789433197] req@ffff9811d2101880 x1876350913702400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433197.350200:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913702400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433197.350211:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433197.350215:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433197.350220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913702400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433197.350227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433194/real 1789433197] req@ffff9811d2101180 x1876350913702528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433197.350239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913702528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433197.350247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433197.350251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433197.350254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433197.350260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433194/real 1789433197] req@ffff9811d2101f80 x1876350913702656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433197.350271:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913702656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433197.350283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433197.350286:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433197.350289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433197.350295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433194/real 1789433197] req@ffff9811d2100e00 x1876350913702784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433197.350306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913702784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433197.350314:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433197.350317:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433197.350321:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913702784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433198.246159:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433199.270044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433199.397951:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433199.397960:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433199.397964:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433199.397972:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5568 00000100:00080000:2.0:1789433199.397979:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433199.397989:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433199.398026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913702912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433199.398054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433199.398057:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433199.398059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433199.398064:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5568 00000100:00080000:2.0:1789433199.398068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433199.398072:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433199.398081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433199.398094:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913703040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433199.398103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433199.398107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433199.398109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433199.398113:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5568 00000100:00080000:2.0:1789433199.398117:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433199.398121:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433199.398128:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913702912:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789433199.398128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433199.398143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913703168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433199.398156:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433199.398160:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433199.398161:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433199.398177:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5568 00000100:00080000:2.0:1789433199.398182:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433199.398188:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433199.398191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789433199.398195:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433199.398209:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913703296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433199.398241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433199/real 1789433199] req@ffff9811d2102a00 x1876350913702912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433199.398260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913702912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433199.398270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433199.398274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433199.398278:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913702912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433199.398285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433199.398312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433199.398325:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433199/real 1789433199] req@ffff9811d2101c00 x1876350913703040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433199.398352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913703040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433199.398360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433199.398363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433199.398374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433199.398397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433199.398415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433199.398427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433199/real 1789433199] req@ffff9811d2103b80 x1876350913703168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433199.398437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913703168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433199.398444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433199.398447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433199.398450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433199.398455:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433199.398474:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433199.398483:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433199/real 1789433199] req@ffff9811d2101500 x1876350913703296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433199.398505:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913703296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433199.398516:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433199.398519:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433199.398521:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433200.294068:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433201.318097:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433202.341989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433203.366027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433204.390246:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433204.518117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433204.518126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433204.518129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433204.518138:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5574 00000100:00080000:3.0:1789433204.518156:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433204.518167:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433204.518202:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913703424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433204.518232:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433204.518236:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433204.518238:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433204.518242:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5574 00000100:00080000:3.0:1789433204.518246:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433204.518251:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433204.518260:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433204.518274:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913703552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433204.518284:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433204.518289:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433204.518291:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433204.518306:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5574 00000100:00080000:3.0:1789433204.518310:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433204.518315:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433204.518319:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433204.518330:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913703680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433204.518340:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433204.518343:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433204.518345:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433204.518348:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5574 00000100:00080000:3.0:1789433204.518351:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433204.518356:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433204.518361:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433204.518371:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913703808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433204.518463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913703424:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433204.518526:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433204.518563:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433204/real 1789433204] req@ffff9811d1bc6300 x1876350913703424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433204.518580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913703424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433204.518590:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433204.518593:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433204.518597:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913703424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433204.518604:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433204.518625:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433204.518637:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433204/real 1789433204] req@ffff9811d1bc5880 x1876350913703552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433204.518649:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913703552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433204.518663:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433204.518666:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433204.518669:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433204.518675:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433204.518693:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433204.518704:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433204/real 1789433204] req@ffff9811d1bc5f80 x1876350913703680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433204.518716:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913703680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433204.518723:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433204.518726:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433204.518729:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433204.518734:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703808:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433204.518754:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433204.518764:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433204/real 1789433204] req@ffff9811d1bc4000 x1876350913703808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433204.518775:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913703808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433204.518783:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433204.518786:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433204.518790:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913703808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433205.414024:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433206.438108:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433207.462005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433208.486051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433209.509974:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433209.637981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433209.637990:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433209.637994:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433209.638002:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5579 00000100:00080000:3.0:1789433209.638009:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433209.638025:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433209.638055:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913703936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433209.638080:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433209.638083:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433209.638085:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433209.638089:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5579 00000100:00080000:3.0:1789433209.638092:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433209.638096:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433209.638104:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433209.638116:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913704064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433209.638127:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433209.638129:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913703936:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433209.638131:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433209.638134:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433209.638141:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5579 00000100:00080000:3.0:1789433209.638144:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433209.638147:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433209.638152:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433209.638161:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913704192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433209.638175:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433209.638178:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433209.638181:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433209.638184:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5579 00000100:00080000:3.0:1789433209.638187:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433209.638193:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433209.638194:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433209.638201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433209.638212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913704320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433209.638235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433209/real 1789433209] req@ffff9811d1bc5180 x1876350913703936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433209.638254:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913703936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433209.638276:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433209.638279:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433209.638283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913703936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433209.638304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433209.638338:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433209.638350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433209/real 1789433209] req@ffff9811d1bc7800 x1876350913704064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433209.638361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913704064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433209.638370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433209.638373:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433209.638376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433209.638382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433209.638397:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433209.638405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433209/real 1789433209] req@ffff9811d1bc5c00 x1876350913704192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433209.638414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913704192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433209.638422:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433209.638425:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433209.638428:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433209.638433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433209.638464:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433209.638475:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433209/real 1789433209] req@ffff9811d1bc6a00 x1876350913704320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433209.638486:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913704320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433209.638494:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433209.638497:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433209.638500:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433210.534052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433211.558010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433212.581946:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433213.606069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433214.629985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433214.757952:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433214.757966:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433214.757970:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433214.757980:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5584 00000100:00080000:1.0:1789433214.757992:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433214.758002:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433214.758036:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913704448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433214.758067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433214.758071:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433214.758075:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433214.758080:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5584 00000100:00080000:1.0:1789433214.758083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433214.758093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433214.758102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433214.758113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913704448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433214.758117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913704576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433214.758135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433214.758138:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433214.758140:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433214.758144:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5584 00000100:00080000:1.0:1789433214.758156:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433214.758165:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433214.758170:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789433214.758172:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433214.758186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913704704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433214.758196:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433214.758199:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433214.758201:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433214.758205:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5584 00000100:00080000:1.0:1789433214.758208:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433214.758213:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433214.758218:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433214.758234:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433214/real 1789433214] req@ffff9811d0bdd180 x1876350913704448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433214.758245:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913704832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433214.758261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913704448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433214.758273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433214.758277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433214.758281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913704448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433214.758300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433214.758320:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433214.758331:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433214/real 1789433214] req@ffff9811d0bdc380 x1876350913704576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433214.758342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913704576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433214.758349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433214.758352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433214.758356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433214.758361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433214.758377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433214.758386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433214/real 1789433214] req@ffff9811d0bddc00 x1876350913704704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433214.758397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913704704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433214.758404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433214.758407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433214.758410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433214.758416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433214.758434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433214.758442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433214/real 1789433214] req@ffff9811d0bdd500 x1876350913704832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433214.758453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913704832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433214.758461:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433214.758464:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433214.758467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913704832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433215.654009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433216.677991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433217.701951:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433218.725985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433219.749988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433219.877947:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433219.877954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433219.877958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433219.877966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5589 00000100:00080000:3.0:1789433219.877972:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433219.877979:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433219.878007:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913704960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433219.878027:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433219.878029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433219.878031:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433219.878033:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5589 00000100:00080000:3.0:1789433219.878035:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433219.878037:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433219.878044:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433219.878053:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913705088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433219.878059:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433219.878067:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433219.878068:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433219.878072:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5589 00000100:00080000:3.0:1789433219.878074:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433219.878077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433219.878080:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433219.878086:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913704960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433219.878088:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913705216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433219.878100:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433219.878102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433219.878103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433219.878105:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5589 00000100:00080000:3.0:1789433219.878108:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433219.878110:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433219.878112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433219.878120:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913705344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433219.878150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433219.878198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433219/real 1789433219] req@ffff9811d1bc7480 x1876350913704960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433219.878216:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913704960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433219.878226:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433219.878230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433219.878233:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913704960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433219.878240:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433219.878261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433219.878272:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433219/real 1789433219] req@ffff9811d1bc6a00 x1876350913705088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433219.878285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913705088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433219.878293:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433219.878295:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433219.878298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433219.878318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433219.878335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433219.878345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433219/real 1789433219] req@ffff9811d1bc5c00 x1876350913705216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433219.878356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913705216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433219.878364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433219.878367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433219.878369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433219.878375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705344:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433219.878391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433219.878401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433219/real 1789433219] req@ffff9811d1bc7800 x1876350913705344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433219.878412:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913705344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433219.878426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433219.878429:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433219.878432:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433220.774012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433221.798042:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433222.822089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433223.846049:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433224.869985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433224.997938:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433224.997947:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433224.997951:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433224.997962:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5594 00000100:00080000:2.0:1789433224.997970:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433224.997980:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433224.998019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913705472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433224.998042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433224.998045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433224.998046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433224.998049:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5594 00000100:00080000:2.0:1789433224.998052:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433224.998055:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433224.998062:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433224.998071:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913705600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433224.998078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433224.998081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433224.998082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433224.998084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5594 00000100:00080000:2.0:1789433224.998086:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433224.998089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433224.998092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433224.998100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913705472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433224.998101:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913705728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433224.998118:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433224.998121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433224.998123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433224.998127:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5594 00000100:00080000:2.0:1789433224.998130:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433224.998135:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433224.998140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433224.998162:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913705856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433224.998179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433224.998226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433224/real 1789433224] req@ffff9811d2102d80 x1876350913705472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433224.998245:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913705472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433224.998255:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433224.998259:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433224.998263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913705472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433224.998270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433224.998294:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433224.998307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433224/real 1789433224] req@ffff9811d2103800 x1876350913705600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433224.998319:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913705600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433224.998327:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433224.998330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433224.998333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433224.998355:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433224.998375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433224.998387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433224/real 1789433224] req@ffff9811d2100000 x1876350913705728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433224.998398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913705728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433224.998405:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433224.998408:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433224.998411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433224.998416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433224.998436:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433224.998447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433224/real 1789433224] req@ffff9811d2101500 x1876350913705856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433224.998458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913705856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433224.998465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433224.998468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433224.998472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913705856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433225.893998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433226.917993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433227.942012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433228.965988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433229.990022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433230.117971:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433230.117981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433230.117985:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433230.117996:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5599 00000100:00080000:1.0:1789433230.118004:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433230.118015:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433230.118048:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913705984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433230.118081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433230.118085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433230.118087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433230.118091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5599 00000100:00080000:1.0:1789433230.118095:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433230.118100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433230.118122:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433230.118136:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913706112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433230.118145:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433230.118151:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433230.118164:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433230.118168:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5599 00000100:00080000:1.0:1789433230.118172:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433230.118176:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433230.118181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433230.118192:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913706240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433230.118202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433230.118205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433230.118207:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433230.118211:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5599 00000100:00080000:1.0:1789433230.118214:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433230.118219:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433230.118225:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433230.118237:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913706368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433230.118241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913705984:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433230.118322:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433230.118573:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433230/real 1789433230] req@ffff9811d0bde680 x1876350913705984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433230.118592:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913705984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433230.118602:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433230.118607:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433230.118610:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913705984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433230.118618:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433230.118642:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433230.118655:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433230/real 1789433230] req@ffff9811d0bdf480 x1876350913706112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433230.118667:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913706112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433230.118675:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433230.118678:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433230.118682:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433230.118688:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433230.118707:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433230.118718:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433230/real 1789433230] req@ffff9811d0bdfb80 x1876350913706240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433230.118729:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913706240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433230.118743:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433230.118746:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433230.118749:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433230.118780:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433230.118801:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433230.118812:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433230/real 1789433230] req@ffff9811d0bdc700 x1876350913706368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433230.118823:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913706368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433230.119410:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433230.119415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433230.119419:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433231.014010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433232.038004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433233.062006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433234.086001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433235.110034:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433235.237934:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433235.237943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433235.237948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433235.237957:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5604 00000100:00080000:1.0:1789433235.237964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433235.237972:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433235.238001:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913706496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433235.238023:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433235.238030:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433235.238032:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433235.238036:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5604 00000100:00080000:1.0:1789433235.238039:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433235.238045:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433235.238054:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433235.238078:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913706496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433235.238080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913706624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433235.238094:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433235.238097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433235.238099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433235.238111:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5604 00000100:00080000:1.0:1789433235.238125:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433235.238131:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433235.238136:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789433235.238138:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433235.238151:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913706752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433235.238162:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433235.238165:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433235.238167:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433235.238172:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433235/real 1789433235] req@ffff9811d0bdea00 x1876350913706496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433235.238172:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5604 00000100:00080000:1.0:1789433235.238180:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433235.238187:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433235.238191:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913706496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433235.238199:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789433235.238200:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789433235.238203:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433235.238208:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913706496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433235.238217:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913706880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433235.238228:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433235.238252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433235.238263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433235/real 1789433235] req@ffff9811d0bded80 x1876350913706624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433235.238275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913706624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433235.238282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433235.238285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433235.238289:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433235.238294:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433235.238312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433235.238322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433235/real 1789433235] req@ffff9811d0bdc000 x1876350913706752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433235.238332:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913706752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433235.238337:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433235.238339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433235.238341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433235.238350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706880:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433235.238363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433235.238369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433235/real 1789433235] req@ffff9811d0bdd880 x1876350913706880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433235.238380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913706880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433235.238387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433235.238390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433235.238393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913706880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433236.133978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433237.157993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433238.182019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433239.205982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433240.230013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433240.357894:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433240.357905:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433240.357908:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433240.357919:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5609 00000100:00080000:3.0:1789433240.357936:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433240.357947:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433240.357986:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913707008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433240.358010:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433240.358014:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433240.358016:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433240.358020:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5609 00000100:00080000:3.0:1789433240.358023:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433240.358029:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433240.358049:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433240.358067:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913707008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433240.358068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913707136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433240.358084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433240.358088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433240.358089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433240.358093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5609 00000100:00080000:3.0:1789433240.358097:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433240.358103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433240.358108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433240.358117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913707264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433240.358124:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433240.358127:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433240.358129:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433240.358131:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5609 00000100:00080000:3.0:1789433240.358133:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433240.358136:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433240.358139:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433240.358141:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433240.358149:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913707392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433240.358188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433240/real 1789433240] req@ffff9811d1bc5180 x1876350913707008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433240.358206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913707008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433240.358216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433240.358219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433240.358222:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913707008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433240.358242:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433240.358263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433240.358274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433240/real 1789433240] req@ffff9811d1bc4000 x1876350913707136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433240.358285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913707136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433240.358298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433240.358301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433240.358305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433240.358310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433240.358327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433240.358336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433240/real 1789433240] req@ffff9811d1bc5f80 x1876350913707264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433240.358346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913707264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433240.358354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433240.358356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433240.358359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433240.358364:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433240.358381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433240.358390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433240/real 1789433240] req@ffff9811d1bc5880 x1876350913707392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433240.358401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913707392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433240.358408:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433240.358410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433240.358413:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433241.254004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433242.278045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433243.302025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433244.326014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433245.350020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433245.477974:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433245.477984:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433245.477989:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433245.478010:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5615 00000100:00080000:3.0:1789433245.478020:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433245.478032:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433245.478073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913707520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433245.478112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433245.478120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433245.478122:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433245.478126:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5615 00000100:00080000:3.0:1789433245.478130:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433245.478135:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433245.478146:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433245.478164:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913707648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433245.478173:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433245.478177:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789433245.478179:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913707520:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433245.478179:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433245.478185:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5615 00000100:00080000:3.0:1789433245.478190:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433245.478193:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433245.478196:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433245.478203:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913707776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433245.478209:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433245.478212:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433245.478213:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433245.478215:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5615 00000100:00080000:3.0:1789433245.478217:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433245.478225:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433245.478231:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433245.478239:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913707904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433245.478263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433245.478327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433245/real 1789433245] req@ffff9811d1bc6300 x1876350913707520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433245.478349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913707520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433245.478361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433245.478365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433245.478369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913707520:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433245.478395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433245.478419:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433245.478433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433245/real 1789433245] req@ffff9811d1bc4700 x1876350913707648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433245.478446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913707648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433245.478454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433245.478457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433245.478461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433245.478467:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433245.478486:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433245.478508:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433245/real 1789433245] req@ffff9811d1bc6680 x1876350913707776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433245.478519:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913707776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433245.478527:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433245.478530:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433245.478533:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433245.478540:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707904:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433245.478561:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433245.478572:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433245/real 1789433245] req@ffff9811d1bc5500 x1876350913707904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433245.478584:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913707904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433245.478593:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433245.478596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433245.478600:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913707904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433246.374069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433247.398019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433248.422218:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433249.446025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433250.470805:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433250.597929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433250.597938:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433250.597943:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433250.597953:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5620 00000100:00080000:3.0:1789433250.597974:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433250.597985:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433250.598016:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913708032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433250.598040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433250.598044:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433250.598046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433250.598050:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5620 00000100:00080000:3.0:1789433250.598054:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433250.598059:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433250.598068:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433250.598084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913708032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433250.598085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913708160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433250.598098:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433250.598100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433250.598101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433250.598105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5620 00000100:00080000:3.0:1789433250.598110:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433250.598115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433250.598120:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433250.598132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913708288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433250.598139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433250.598175:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433250/real 1789433250] req@ffff9811d1bc7100 x1876350913708032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433250.598192:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913708032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433250.598203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433250.598206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433250.598210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913708032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433250.598230:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433250.598252:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433250.598264:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433250/real 1789433250] req@ffff9811d1bc5500 x1876350913708160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433250.598276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913708160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433250.598284:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433250.598287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433250.598290:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708160:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789433250.598466:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433250.598469:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433250.598471:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433250.598475:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5620 00000100:00080000:3.0:1789433250.598479:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433250.598485:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433250.598491:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433250.598504:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433250.598506:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913708416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433250.598540:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433250.598554:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433250/real 1789433250] req@ffff9811d1bc6680 x1876350913708288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433250.598567:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913708288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433250.598575:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433250.598578:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433250.598582:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433250.598595:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433250.598616:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433250.598627:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433250/real 1789433250] req@ffff9811d1bc4700 x1876350913708416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433250.598639:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913708416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433250.598647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433250.598650:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433250.598653:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433251.494044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433252.518030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433253.542060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433254.566053:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433255.590075:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433255.718586:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433255.718596:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433255.718600:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433255.718609:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5625 00000100:00080000:3.0:1789433255.718618:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433255.718628:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433255.718663:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913708544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433255.718688:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433255.718692:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433255.718694:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433255.718698:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5625 00000100:00080000:3.0:1789433255.718701:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433255.718706:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433255.718716:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433255.718730:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913708672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433255.718736:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913708544:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433255.718742:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433255.718746:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433255.718748:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433255.718751:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5625 00000100:00080000:3.0:1789433255.718755:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433255.718760:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433255.718765:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433255.718776:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913708800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433255.718786:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433255.718789:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433255.718791:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433255.718794:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5625 00000100:00080000:3.0:1789433255.718798:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433255.718817:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433255.718818:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433255.718829:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433255.718874:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913708928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433255.718880:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433255/real 1789433255] req@ffff9811d1bc6300 x1876350913708544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433255.718901:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913708544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433255.718911:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433255.718915:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433255.718931:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913708544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433255.718951:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433255.718972:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433255.718984:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433255/real 1789433255] req@ffff9811d1bc5880 x1876350913708672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433255.718997:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913708672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433255.719005:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433255.719008:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433255.719011:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433255.719017:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433255.719033:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433255.719043:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433255/real 1789433255] req@ffff9811d1bc5f80 x1876350913708800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433255.719054:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913708800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433255.719061:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433255.719064:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433255.719076:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433255.719082:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433255.719100:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433255.719110:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433255/real 1789433255] req@ffff9811d1bc4000 x1876350913708928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433255.719122:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913708928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433255.719137:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433255.719140:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433255.719143:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913708928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433256.614005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433257.637995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433258.662457:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433259.686012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433260.709968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433260.837931:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433260.837942:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433260.837945:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433260.837955:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5630 00000100:00080000:3.0:1789433260.837964:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433260.837974:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433260.838003:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913709056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433260.838024:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433260.838027:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433260.838028:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433260.838031:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5630 00000100:00080000:3.0:1789433260.838033:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433260.838036:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433260.838043:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433260.838052:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913709184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433260.838063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433260.838066:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433260.838068:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433260.838073:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5630 00000100:00080000:3.0:1789433260.838076:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433260.838081:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433260.838086:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433260.838096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913709312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433260.838108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433260.838114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1789433260.838116:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913709056:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433260.838117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433260.838126:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5630 00000100:00080000:3.0:1789433260.838137:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433260.838142:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433260.838146:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433260.838154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913709440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433260.838265:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433260.838316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433260.838344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433261.734010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433262.758052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433263.782121:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789433263.910218:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789433263.910274:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789433263.910295:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433263.910338:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433263.910348:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789433263.910356:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433263.910396:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433260/real 1789433263] req@ffff9811d1bc5c00 x1876350913709312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433263.910426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913709312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433263.910440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433263.910445:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433263.910450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433263.910471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433260/real 1789433263] req@ffff9811d1bc6a00 x1876350913709440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433263.910484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913709440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433263.910493:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433263.910498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433263.910502:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709440:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433263.910509:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433260/real 1789433263] req@ffff9811d1bc5180 x1876350913709056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433263.910523:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913709056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433263.910532:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433263.910536:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433263.910539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913709056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433263.910547:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433260/real 1789433263] req@ffff9811d1bc7800 x1876350913709184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433263.910559:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913709184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433263.910568:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433263.910571:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433263.910574:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709184:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789433264.806078:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433265.830043:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433265.958076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433265.958086:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433265.958089:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433265.958098:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5635 00000100:00080000:1.0:1789433265.958125:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433265.958137:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433265.958196:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913709568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433265.958234:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433265.958238:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433265.958240:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433265.958244:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5635 00000100:00080000:1.0:1789433265.958248:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433265.958269:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433265.958279:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433265.958304:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913709696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433265.958314:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433265.958317:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433265.958319:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433265.958322:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5635 00000100:00080000:1.0:1789433265.958326:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433265.958344:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433265.958349:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433265.958359:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913709824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433265.958387:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433265.958400:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433265.958402:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433265.958406:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5635 00000100:00080000:1.0:1789433265.958409:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433265.958413:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433265.958418:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433265.958462:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913709952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433265.958498:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913709568:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433265.958564:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433265.958610:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433265/real 1789433265] req@ffff9811d0bdf800 x1876350913709568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433265.958625:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913709568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433265.958634:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433265.958637:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433265.958641:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913709568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433265.958647:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433265.958667:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433265.958678:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433265/real 1789433265] req@ffff9811d0bde300 x1876350913709696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433265.958687:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913709696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433265.958694:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433265.958696:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433265.958698:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433265.958704:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433265.958720:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433265.958730:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433265/real 1789433265] req@ffff9811d0bdf100 x1876350913709824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433265.958741:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913709824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433265.958748:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433265.958750:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433265.958753:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433265.958759:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433265.958777:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433265.958787:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433265/real 1789433265] req@ffff9811d0bdd880 x1876350913709952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433265.958806:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913709952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433265.958814:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433265.958816:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433265.958819:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913709952:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433266.854011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433267.878004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433268.902015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433269.926004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433270.950064:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433271.077963:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433271.077970:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433271.077974:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433271.077981:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5640 00000100:00080000:3.0:1789433271.077994:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433271.078005:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433271.078037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913710080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433271.078062:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433271.078064:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433271.078065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433271.078068:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5640 00000100:00080000:3.0:1789433271.078071:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433271.078074:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433271.078081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433271.078090:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913710208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433271.078097:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433271.078099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433271.078101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433271.078104:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5640 00000100:00080000:3.0:1789433271.078108:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433271.078112:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433271.078118:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433271.078119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913710080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433271.078135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913710336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433271.078153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433271.078156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433271.078157:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433271.078160:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5640 00000100:00080000:3.0:1789433271.078163:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433271.078169:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433271.078173:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433271.078178:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433271.078183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913710464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433271.078218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433271/real 1789433271] req@ffff9811d1bc7480 x1876350913710080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433271.078233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913710080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433271.078244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433271.078248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433271.078251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913710080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433271.078257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433271.078278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433271.078289:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433271/real 1789433271] req@ffff9811d1bc7800 x1876350913710208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433271.078300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913710208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433271.078307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433271.078311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433271.078314:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433271.078333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433271.078352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433271.078363:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433271/real 1789433271] req@ffff9811d1bc5180 x1876350913710336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433271.078374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913710336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433271.078381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433271.078384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433271.078387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433271.078392:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710464:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433271.078410:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433271.078420:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433271/real 1789433271] req@ffff9811d1bc6a00 x1876350913710464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433271.078430:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913710464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433271.078444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433271.078447:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433271.078450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710464:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433271.974051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433272.998000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433274.022235:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433275.046026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433276.070004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433276.197957:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433276.197966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433276.197970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433276.197979:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5645 00000100:00080000:3.0:1789433276.197988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433276.198001:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433276.198037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913710592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433276.198064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433276.198068:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433276.198070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433276.198074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5645 00000100:00080000:3.0:1789433276.198078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433276.198084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433276.198094:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433276.198109:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913710592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433276.198111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913710720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433276.198124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433276.198126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433276.198127:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433276.198130:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5645 00000100:00080000:3.0:1789433276.198133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433276.198136:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433276.198139:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433276.198148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913710848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433276.198154:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433276.198156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433276.198157:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433276.198159:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5645 00000100:00080000:3.0:1789433276.198163:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433276.198166:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433276.198168:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433276.198173:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433276.198177:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913710976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433276.198213:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433276/real 1789433276] req@ffff9811d1bc5c00 x1876350913710592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433276.198230:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913710592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433276.198237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433276.198239:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433276.198241:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913710592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433276.198260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433276.198287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433276.198299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433276/real 1789433276] req@ffff9811d1bc4000 x1876350913710720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433276.198310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913710720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433276.198318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433276.198321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433276.198324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433276.198330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433276.198348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433276.198358:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433276/real 1789433276] req@ffff9811d1bc5f80 x1876350913710848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433276.198370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913710848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433276.198377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433276.198380:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433276.198383:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433276.198387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710976:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433276.198406:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433276.198416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433276/real 1789433276] req@ffff9811d1bc5880 x1876350913710976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433276.198428:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913710976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433276.198436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433276.198438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433276.198441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913710976:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433277.093988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433278.117955:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433279.143080:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433280.166009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433281.190030:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433281.317936:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433281.317946:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433281.317950:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433281.317959:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5650 00000100:00080000:2.0:1789433281.317968:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433281.317979:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433281.318020:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913711104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433281.318045:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433281.318050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433281.318052:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433281.318056:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5650 00000100:00080000:2.0:1789433281.318060:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433281.318065:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433281.318074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433281.318088:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913711232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433281.318100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913711104:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433281.318100:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433281.318108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433281.318111:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433281.318115:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5650 00000100:00080000:2.0:1789433281.318118:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433281.318122:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433281.318127:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433281.318148:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913711360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433281.318159:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433281.318163:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433281.318165:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433281.318170:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433281.318172:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5650 00000100:00080000:2.0:1789433281.318179:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433281.318186:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433281.318192:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433281.318205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913711488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433281.318214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433281/real 1789433281] req@ffff9811d2103b80 x1876350913711104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433281.318235:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913711104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433281.318246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433281.318251:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433281.318254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913711104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433281.318275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433281.318298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433281.318310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433281/real 1789433281] req@ffff9811d2101c00 x1876350913711232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433281.318324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913711232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433281.318331:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433281.318335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433281.318338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433281.318344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433281.318363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433281.318373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433281/real 1789433281] req@ffff9811d2102a00 x1876350913711360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433281.318384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913711360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433281.318412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433281.318416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433281.318419:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433281.318425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711488:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433281.318448:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433281.318459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433281/real 1789433281] req@ffff9811d2100e00 x1876350913711488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433281.318475:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913711488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433281.318483:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433281.318485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433281.318488:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711488:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433282.214005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433283.238006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433284.261998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433285.285992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433286.309980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433286.438012:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433286.438019:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433286.438022:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433286.438030:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5656 00000100:00080000:3.0:1789433286.438036:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433286.438043:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433286.438073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913711616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433286.438093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433286.438095:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433286.438096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433286.438099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5656 00000100:00080000:3.0:1789433286.438102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433286.438106:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433286.438115:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433286.438127:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913711744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433286.438134:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913711616:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433286.438144:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433286.438148:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433286.438149:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433286.438154:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5656 00000100:00080000:3.0:1789433286.438157:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433286.438161:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433286.438165:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433286.438174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913711872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433286.438181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433286.438183:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433286.438184:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433286.438187:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5656 00000100:00080000:3.0:1789433286.438189:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433286.438192:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433286.438197:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789433286.438199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433286.438207:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913712000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433286.438240:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433286/real 1789433286] req@ffff9811d1bc6300 x1876350913711616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433286.438258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913711616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433286.438265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433286.438267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433286.438270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913711616:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433286.438286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433286.438303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433286.438315:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433286/real 1789433286] req@ffff9811d1bc4700 x1876350913711744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433286.438327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913711744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433286.438335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433286.438338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433286.438341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433286.438347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433286.438364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433286.438371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433286/real 1789433286] req@ffff9811d1bc6680 x1876350913711872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433286.438381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913711872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433286.438387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433286.438390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433286.438393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913711872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433286.438398:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712000:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433286.438416:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433286.438427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433286/real 1789433286] req@ffff9811d1bc5500 x1876350913712000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433286.438435:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913712000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433286.438440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433286.438442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433286.438443:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712000:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433287.334109:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433288.358656:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433289.383472:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433290.406000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433291.430001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433291.557940:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433291.557948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433291.557951:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433291.557959:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5661 00000100:00080000:2.0:1789433291.557966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433291.557974:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433291.558004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913712128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433291.558031:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433291.558035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433291.558036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433291.558041:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5661 00000100:00080000:2.0:1789433291.558044:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433291.558049:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433291.558059:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433291.558073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913712256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433291.558085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433291.558087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913712128:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433291.558091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433291.558095:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433291.558099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5661 00000100:00080000:2.0:1789433291.558102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433291.558107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433291.558111:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433291.558126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913712384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433291.558136:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433291.558139:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433291.558140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433291.558144:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433291.558148:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5661 00000100:00080000:2.0:1789433291.558153:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433291.558158:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433291.558163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433291.558175:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913712512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433291.558190:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433291/real 1789433291] req@ffff9811d2101f80 x1876350913712128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433291.558208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913712128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433291.558218:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433291.558222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433291.558226:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913712128:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433291.558247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433291.558268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433291.558280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433291/real 1789433291] req@ffff9811d2101180 x1876350913712256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433291.558292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913712256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433291.558297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433291.558299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433291.558301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433291.558305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433291.558317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433291.558322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433291/real 1789433291] req@ffff9811d2101880 x1876350913712384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433291.558329:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913712384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433291.558333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433291.558335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433291.558337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433291.558340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712512:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433291.558352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433291.558360:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433291/real 1789433291] req@ffff9811d2100e00 x1876350913712512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433291.558370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913712512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433291.558383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433291.558386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433291.558389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712512:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433292.454003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433293.477979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433294.502000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433295.526074:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433296.549985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433296.677913:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433296.677923:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433296.677927:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433296.677935:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5666 00000100:00080000:2.0:1789433296.677944:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433296.677956:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433296.677990:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913712640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433296.678020:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433296.678024:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433296.678025:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433296.678029:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5666 00000100:00080000:2.0:1789433296.678033:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433296.678038:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433296.678047:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433296.678058:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913712768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433296.678067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433296.678069:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433296.678071:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433296.678074:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5666 00000100:00080000:2.0:1789433296.678077:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433296.678081:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433296.678086:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433296.678098:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913712896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433296.678109:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433296.678113:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433296.678117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433296.678118:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913712640:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433296.678124:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5666 00000100:00080000:2.0:1789433296.678128:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433296.678132:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433296.678136:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433296.678146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913713024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433296.678181:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433296.678210:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433296/real 1789433296] req@ffff9811d2102a00 x1876350913712640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433296.678226:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913712640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433296.678236:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433296.678240:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433296.678244:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913712640:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433296.678250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433296.678272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433296.678282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433296/real 1789433296] req@ffff9811d2101c00 x1876350913712768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433296.678290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913712768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433296.678295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433296.678298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433296.678300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433296.678318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433296.678336:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433296.678346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433296/real 1789433296] req@ffff9811d2103b80 x1876350913712896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433296.678358:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913712896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433296.678365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433296.678367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433296.678369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913712896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433296.678372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713024:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433296.678386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433296.678392:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433296/real 1789433296] req@ffff9811d2101500 x1876350913713024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433296.678399:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913713024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433296.678404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433296.678407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433296.678410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713024:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433297.574001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433298.597996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433299.622001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433300.645994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433301.670012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433301.797986:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433301.797996:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433301.798000:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433301.798010:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5671 00000100:00080000:3.0:1789433301.798018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433301.798030:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433301.798063:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913713152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433301.798088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433301.798092:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433301.798094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433301.798098:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5671 00000100:00080000:3.0:1789433301.798102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433301.798108:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433301.798117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433301.798122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913713152:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433301.798134:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913713280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433301.798146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433301.798149:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433301.798151:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433301.798154:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5671 00000100:00080000:3.0:1789433301.798159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433301.798164:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433301.798169:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433301.798181:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913713408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433301.798184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433301.798196:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433301.798200:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433301.798203:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433301.798207:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5671 00000100:00080000:3.0:1789433301.798211:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433301.798216:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433301.798221:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433301.798226:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433301/real 1789433301] req@ffff9811d1bc7100 x1876350913713152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433301.798235:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913713536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433301.798250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913713152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433301.798262:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433301.798267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433301.798271:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913713152:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433301.798292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433301.798314:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433301.798327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433301/real 1789433301] req@ffff9811d1bc5500 x1876350913713280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433301.798338:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913713280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433301.798346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433301.798349:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433301.798352:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433301.798358:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433301.798376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433301.798387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433301/real 1789433301] req@ffff9811d1bc6680 x1876350913713408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433301.798405:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913713408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433301.798412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433301.798415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433301.798418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433301.798423:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713536:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433301.798441:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433301.798470:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433301/real 1789433301] req@ffff9811d1bc4700 x1876350913713536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433301.798482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913713536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433301.798490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433301.798492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433301.798495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713536:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433302.693995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433303.718002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433304.742062:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433305.766052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433306.789992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433306.918040:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433306.918050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433306.918055:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433306.918065:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5676 00000100:00080000:1.0:1789433306.918074:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433306.918093:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433306.918124:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913713664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433306.918146:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433306.918148:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433306.918149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433306.918152:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5676 00000100:00080000:1.0:1789433306.918155:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433306.918158:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433306.918165:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433306.918173:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913713792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433306.918179:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433306.918181:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433306.918182:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433306.918185:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5676 00000100:00080000:1.0:1789433306.918186:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433306.918188:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433306.918191:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433306.918195:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913713920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433306.918204:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913713664:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433306.918205:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433306.918214:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433306.918215:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433306.918219:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5676 00000100:00080000:1.0:1789433306.918222:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433306.918227:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433306.918231:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433306.918244:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913714048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433306.918269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433306.918306:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433306/real 1789433306] req@ffff9811d0bdc000 x1876350913713664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433306.918323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913713664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433306.918334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433306.918338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433306.918342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913713664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433306.918349:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433306.918370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433306.918382:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433306/real 1789433306] req@ffff9811d0bded80 x1876350913713792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433306.918401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913713792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433306.918410:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433306.918413:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433306.918416:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433306.918442:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433306.918462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433306.918472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433306/real 1789433306] req@ffff9811d0bdea00 x1876350913713920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433306.918483:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913713920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433306.918490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433306.918494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433306.918497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913713920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433306.918503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714048:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433306.918521:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433306.918532:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433306/real 1789433306] req@ffff9811d0bdc700 x1876350913714048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433306.918558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913714048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433306.918567:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433306.918570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433306.918573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714048:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433307.814015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433308.837993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433309.862068:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433310.885983:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433311.909998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433312.037935:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433312.037944:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433312.037948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433312.037957:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5681 00000100:00080000:1.0:1789433312.037964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433312.037973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433312.038000:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913714176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433312.038018:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433312.038020:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433312.038022:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433312.038024:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5681 00000100:00080000:1.0:1789433312.038026:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433312.038034:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433312.038042:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433312.038051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913714304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433312.038061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433312.038065:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913714176:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433312.038066:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433312.038071:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433312.038077:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5681 00000100:00080000:1.0:1789433312.038080:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433312.038085:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433312.038090:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433312.038101:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913714432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433312.038114:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433312.038115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433312.038119:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433312.038121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433312.038124:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5681 00000100:00080000:1.0:1789433312.038127:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433312.038132:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433312.038135:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433312.038145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913714560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433312.038149:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433312/real 1789433312] req@ffff9811d0bdfb80 x1876350913714176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433312.038175:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913714176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433312.038189:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433312.038191:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433312.038194:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913714176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433312.038209:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433312.038224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433312.038231:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433312/real 1789433312] req@ffff9811d0bdf480 x1876350913714304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433312.038239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913714304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433312.038244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433312.038246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433312.038247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433312.038250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433312.038265:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433312.038271:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433312/real 1789433312] req@ffff9811d0bde680 x1876350913714432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433312.038277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913714432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433312.038282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433312.038284:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433312.038286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433312.038288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714560:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433312.038298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433312.038302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433312/real 1789433312] req@ffff9811d0bdd500 x1876350913714560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433312.038312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913714560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433312.038317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433312.038319:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433312.038320:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433312.933990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433313.957990:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433314.982025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433316.005999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433317.030092:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433317.157989:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433317.157998:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433317.158003:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433317.158012:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5686 00000100:00080000:3.0:1789433317.158021:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433317.158029:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433317.158062:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913714688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433317.158092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433317.158095:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433317.158097:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433317.158101:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5686 00000100:00080000:3.0:1789433317.158104:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433317.158109:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433317.158118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433317.158130:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913714816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433317.158140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433317.158143:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433317.158144:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433317.158147:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5686 00000100:00080000:3.0:1789433317.158150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433317.158154:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433317.158158:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433317.158168:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913714944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433317.158179:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433317.158182:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433317.158184:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433317.158187:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5686 00000100:00080000:3.0:1789433317.158195:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789433317.158196:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913714688:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433317.158205:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433317.158212:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433317.158230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913715072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433317.158279:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433317.158328:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433317/real 1789433317] req@ffff9811d1bc6300 x1876350913714688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433317.158346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913714688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433317.158359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433317.158363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433317.158367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913714688:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433317.158375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433317.158397:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433317.158411:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433317/real 1789433317] req@ffff9811d1bc5880 x1876350913714816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433317.158421:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913714816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433317.158428:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433317.158431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433317.158434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433317.158440:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433317.158457:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433317.158584:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433317/real 1789433317] req@ffff9811d1bc5f80 x1876350913714944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433317.158593:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913714944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433317.158598:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433317.158601:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433317.158603:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913714944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433317.158631:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715072:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433317.158654:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433317.158662:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433317/real 1789433317] req@ffff9811d1bc4000 x1876350913715072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433317.158669:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913715072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433317.158675:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433317.158677:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433317.158679:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715072:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433318.053944:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433319.077992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433320.101986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433321.126007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433322.149986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433322.277959:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433322.277968:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433322.277976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433322.277986:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5691 00000100:00080000:2.0:1789433322.277994:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433322.278004:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433322.278036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913715200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433322.278070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433322.278074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433322.278075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433322.278079:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5691 00000100:00080000:2.0:1789433322.278083:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433322.278090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433322.278099:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433322.278107:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913715200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433322.278113:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913715328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433322.278131:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433322.278134:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433322.278136:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433322.278140:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5691 00000100:00080000:2.0:1789433322.278144:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433322.278149:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433322.278154:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789433322.278164:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433322.278166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913715456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433322.278176:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433322.278179:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433322.278181:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433322.278185:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5691 00000100:00080000:2.0:1789433322.278189:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433322.278194:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433322.278199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433322.278206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433322/real 1789433322] req@ffff9811d2100000 x1876350913715200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789433322.278212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913715584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433322.278232:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913715200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433322.278242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433322.278246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433322.278250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913715200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433322.278269:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433322.278288:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433322.278299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433322/real 1789433322] req@ffff9811d2103800 x1876350913715328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433322.278309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913715328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433322.278317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433322.278320:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433322.278322:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433322.278330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433322.278345:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433322.278353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433322/real 1789433322] req@ffff9811d2102d80 x1876350913715456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433322.278363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913715456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433322.278369:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433322.278372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433322.278375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433322.278380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715584:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433322.278394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433322.278403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433322/real 1789433322] req@ffff9811d2101500 x1876350913715584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433322.278413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913715584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433322.278421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433322.278423:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433322.278440:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715584:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433323.173999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433324.197980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433325.221986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433326.245994:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433327.270004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433327.397960:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433327.397971:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433327.397976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433327.397986:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5696 00000100:00080000:3.0:1789433327.397999:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433327.398011:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433327.398044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913715712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433327.398070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433327.398073:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433327.398074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433327.398078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5696 00000100:00080000:3.0:1789433327.398082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433327.398086:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433327.398096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433327.398111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913715840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433327.398118:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913715712:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433327.398126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433327.398129:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433327.398130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433327.398133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5696 00000100:00080000:3.0:1789433327.398136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433327.398140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433327.398144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433327.398154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913715968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433327.398161:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433327.398164:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433327.398165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433327.398167:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5696 00000100:00080000:3.0:1789433327.398169:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433327.398172:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433327.398175:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433327.398182:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913716096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433327.398211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433327.398238:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433327.398255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716096:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433328.293985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433329.318018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433330.341968:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433330.470201:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433330.470242:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433330.470257:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433330.470298:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433330.470309:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433330.470316:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433330.470402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433327/real 1789433330] req@ffff9811d1bc6a00 x1876350913715840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433330.470426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913715840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433330.470438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433330.470443:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433330.470447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433330.470454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433327/real 1789433330] req@ffff9811d1bc5180 x1876350913715968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433330.470479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913715968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433330.470487:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433330.470509:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433330.470512:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913715968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433330.470520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433327/real 1789433330] req@ffff9811d1bc7800 x1876350913716096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433330.470532:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913716096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433330.470540:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433330.470543:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433330.470547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716096:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433330.470553:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433327/real 1789433330] req@ffff9811d1bc5c00 x1876350913715712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433330.470564:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913715712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433330.470572:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433330.470575:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433330.470578:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913715712:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789433331.365993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433332.389988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433332.517942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433332.517950:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433332.517953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433332.517960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5702 00000100:00080000:3.0:1789433332.517969:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433332.517980:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433332.518009:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913716224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433332.518037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433332.518041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433332.518043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433332.518047:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5702 00000100:00080000:3.0:1789433332.518050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433332.518054:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433332.518063:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433332.518076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913716352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433332.518085:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913716224:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433332.518090:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433332.518095:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433332.518096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433332.518100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5702 00000100:00080000:3.0:1789433332.518105:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433332.518110:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433332.518115:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433332.518139:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913716480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433332.518146:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433332.518158:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433332.518162:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433332.518165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433332.518169:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5702 00000100:00080000:3.0:1789433332.518173:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433332.518178:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433332.518185:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433332.518186:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433332/real 1789433332] req@ffff9811d1bc5c00 x1876350913716224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433332.518201:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913716608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433332.518209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913716224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433332.518230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433332.518235:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433332.518239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913716224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433332.518261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433332.518284:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433332.518297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433332/real 1789433332] req@ffff9811d1bc7800 x1876350913716352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433332.518310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913716352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433332.518318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433332.518322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433332.518325:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433332.518332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433332.518350:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433332.518373:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433332/real 1789433332] req@ffff9811d1bc5180 x1876350913716480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433332.518387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913716480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433332.518396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433332.518399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433332.518402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433332.518408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716608:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433332.518430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433332.518440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433332/real 1789433332] req@ffff9811d1bc6a00 x1876350913716608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433332.518458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913716608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433332.518466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433332.518469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433332.518472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716608:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433333.414018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433334.437963:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433335.461972:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433336.486126:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433337.510005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433337.637976:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433337.638003:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433337.638007:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433337.638016:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5707 00000100:00080000:0.0:1789433337.638025:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433337.638036:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433337.638072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913716736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433337.638101:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433337.638105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433337.638107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433337.638112:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5707 00000100:00080000:0.0:1789433337.638116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433337.638122:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433337.638131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433337.638137:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913716736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433337.638147:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913716864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433337.638159:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433337.638162:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433337.638164:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433337.638168:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5707 00000100:00080000:0.0:1789433337.638171:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433337.638177:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789433337.638198:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789433337.638221:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433337.638238:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433337/real 1789433337] req@ffff9811c8cd6300 x1876350913716736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433337.638239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913716992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433337.638261:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433337.638264:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433337.638267:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433337.638273:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5707 00000100:00080000:0.0:1789433337.638277:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433337.638281:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913716736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433337.638286:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433337.638296:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789433337.638299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433337.638305:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433337.638312:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913717120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433337.638312:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913716736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433337.638363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433337.638396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433337.638418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433337/real 1789433337] req@ffff9811c8cd7b80 x1876350913716864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433337.638431:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913716864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433337.638440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433337.638443:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433337.638446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433337.638452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433337.638469:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433337.638477:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433337/real 1789433337] req@ffff9811c8cd6a00 x1876350913716992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433337.638484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913716992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433337.638490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433337.638493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433337.638497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913716992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433337.638502:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717120:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433337.638532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433337.638546:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433337/real 1789433337] req@ffff9811c8cd4a80 x1876350913717120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433337.638559:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913717120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433337.638564:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433337.638566:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433337.638568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717120:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433338.534019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433339.558089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433340.581989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433341.606017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433342.629987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433342.757927:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433342.757935:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433342.757938:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433342.757948:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5712 00000100:00080000:0.0:1789433342.757956:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433342.757967:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433342.758001:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913717248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433342.758022:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433342.758025:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433342.758026:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433342.758029:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5712 00000100:00080000:0.0:1789433342.758033:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433342.758038:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433342.758048:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433342.758063:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913717376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433342.758063:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913717248:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433342.758079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433342.758082:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433342.758084:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433342.758088:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5712 00000100:00080000:0.0:1789433342.758091:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433342.758096:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433342.758100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433342.758108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913717504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433342.758115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433342.758117:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433342.758118:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433342.758120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5712 00000100:00080000:0.0:1789433342.758122:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433342.758127:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789433342.758127:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789433342.758134:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433342.758142:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913717632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433342.758170:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433342/real 1789433342] req@ffff9811c8cd5f80 x1876350913717248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433342.758182:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913717248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433342.758190:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433342.758192:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433342.758195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913717248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433342.758211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433342.758225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433342.758232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433342/real 1789433342] req@ffff9811c8cd6d80 x1876350913717376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433342.758239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913717376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433342.758248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433342.758250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433342.758253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433342.758259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433342.758285:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433342.758297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433342/real 1789433342] req@ffff9811c8cd5500 x1876350913717504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433342.758308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913717504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433342.758316:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433342.758319:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433342.758322:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433342.758327:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717632:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433342.758347:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433342.758357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433342/real 1789433342] req@ffff9811c8cd4a80 x1876350913717632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433342.758368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913717632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433342.758373:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433342.758375:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433342.758377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717632:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433343.654327:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433344.678146:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433345.702007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433346.727750:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433347.750571:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433347.877984:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433347.877995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433347.878000:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433347.878011:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5717 00000100:00080000:0.0:1789433347.878021:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433347.878031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433347.878064:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913717760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433347.878091:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433347.878094:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433347.878097:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433347.878100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5717 00000100:00080000:0.0:1789433347.878103:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433347.878107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433347.878116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433347.878127:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913717888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433347.878137:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433347.878140:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433347.878142:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433347.878147:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5717 00000100:00100000:3.0:1789433347.878149:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913717760:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433347.878151:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433347.878158:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433347.878163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433347.878174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913718016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433347.878183:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433347.878187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433347.878188:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433347.878192:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5717 00000100:00080000:0.0:1789433347.878195:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433347.878199:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433347.878203:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789433347.878212:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433347.878217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913718144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433347.878263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433347/real 1789433347] req@ffff9811c8cd6a00 x1876350913717760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433347.878282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913717760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433347.878292:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433347.878296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433347.878300:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913717760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433347.878322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433347.878342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433347.878353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433347/real 1789433347] req@ffff9811c8cd7b80 x1876350913717888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433347.878365:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913717888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433347.878373:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433347.878376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433347.878379:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913717888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433347.878384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433347.878398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433347.878406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433347/real 1789433347] req@ffff9811c8cd6300 x1876350913718016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433347.878416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913718016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433347.878423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433347.878426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433347.878429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433347.878434:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718144:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433347.878452:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433347.878463:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433347/real 1789433347] req@ffff9811c8cd4000 x1876350913718144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433347.878474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913718144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433347.878479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433347.878481:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433347.878484:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718144:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433348.774179:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433349.798070:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433350.823069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433351.846257:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433352.870085:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433352.998005:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433352.998012:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433352.998015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433352.998023:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5722 00000100:00080000:3.0:1789433352.998031:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433352.998042:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433352.998075:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913718272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433352.998133:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913718272:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433352.998191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433352.998241:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433352.998252:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433352.998254:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433352.998259:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5722 00000100:00080000:3.0:1789433352.998264:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433352.998269:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433352.998278:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433352.998292:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913718400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433352.998299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433352/real 1789433352] req@ffff9811d1bc7480 x1876350913718272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433352.998302:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433352.998306:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433352.998308:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433352.998311:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5722 00000100:00100000:0.0:1789433352.998314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913718272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433352.998314:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433352.998318:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433352.998322:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789433352.998324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433352.998328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433352.998331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913718272:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433352.998331:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913718528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433352.998339:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433352.998342:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433352.998344:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433352.998347:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5722 00000100:00100000:0.0:1789433352.998350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718400:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789433352.998350:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433352.998354:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433352.998358:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433352.998366:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913718656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433352.998370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433352.998381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433352/real 1789433352] req@ffff9811d1bc6a00 x1876350913718400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433352.998392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913718400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433352.998400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433352.998403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433352.998406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433352.998413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433352.998432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433352.998442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433352/real 1789433352] req@ffff9811d1bc5180 x1876350913718528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433352.998453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913718528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433352.998461:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433352.998463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433352.998467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433352.998479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718656:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433352.998499:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433352.998509:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433352/real 1789433352] req@ffff9811d1bc7800 x1876350913718656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433352.998521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913718656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433352.998529:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433352.998532:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433352.998534:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718656:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433353.894003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433354.918003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433355.942192:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433356.965995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433357.990004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433358.118014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433358.118023:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433358.118026:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433358.118035:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5727 00000100:00080000:0.0:1789433358.118043:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433358.118053:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433358.118087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913718784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433358.118112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433358.118115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433358.118117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433358.118121:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5727 00000100:00080000:0.0:1789433358.118124:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433358.118129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433358.118138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433358.118148:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913718784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433358.118152:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913718912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433358.118174:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433358.118178:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433358.118180:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433358.118183:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5727 00000100:00080000:0.0:1789433358.118187:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433358.118192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433358.118196:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433358.118208:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913719040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433358.118218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433358.118222:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433358.118226:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433358.118228:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433358.118232:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5727 00000100:00080000:0.0:1789433358.118237:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433358.118242:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433358.118247:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433358.118258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433358/real 1789433358] req@ffff9811c8cd5180 x1876350913718784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433358.118260:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913719168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433358.118292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913718784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433358.118304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433358.118309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433358.118313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913718784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433358.118333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433358.118352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433358.118362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433358/real 1789433358] req@ffff9811c8cd7800 x1876350913718912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433358.118374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913718912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433358.118387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433358.118391:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433358.118394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913718912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433358.118399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433358.118414:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433358.118422:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433358/real 1789433358] req@ffff9811c8cd7480 x1876350913719040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433358.118432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913719040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433358.118440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433358.118442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433358.118445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433358.118450:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719168:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433358.118464:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433358.118472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433358/real 1789433358] req@ffff9811c8cd7100 x1876350913719168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433358.118482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913719168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433358.118489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433358.118492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433358.118495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719168:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433359.015032:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433360.038008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433361.062040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433362.086044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433363.109979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433363.237986:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433363.237994:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433363.237997:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433363.238006:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5732 00000100:00080000:3.0:1789433363.238015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433363.238025:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433363.238056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913719296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433363.238077:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433363.238080:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433363.238081:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433363.238084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5732 00000100:00080000:3.0:1789433363.238087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433363.238089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433363.238095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433363.238104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913719424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433363.238110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433363.238113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433363.238114:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433363.238116:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5732 00000100:00080000:3.0:1789433363.238119:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433363.238134:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433363.238140:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913719296:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433363.238141:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433363.238156:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913719552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433363.238168:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433363.238172:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433363.238174:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433363.238177:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5732 00000100:00080000:3.0:1789433363.238183:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433363.238187:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433363.238191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433363.238203:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913719680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433363.238205:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433363.238249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433363/real 1789433363] req@ffff9811d1bc5c00 x1876350913719296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433363.238267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913719296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433363.238277:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433363.238281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433363.238285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913719296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433363.238295:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433363.238316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433363.238327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433363/real 1789433363] req@ffff9811d1bc4000 x1876350913719424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433363.238340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913719424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433363.238347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433363.238351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433363.238354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433363.238376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433363.238393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433363.238403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433363/real 1789433363] req@ffff9811d1bc5f80 x1876350913719552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433363.238414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913719552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433363.238480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433363.238485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433363.238489:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433363.238495:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719680:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433363.238519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433363.238530:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433363/real 1789433363] req@ffff9811d1bc5880 x1876350913719680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433363.238543:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913719680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433363.238551:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433363.238553:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433363.238556:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719680:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433364.133980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433365.159901:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433366.182017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433367.206023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433368.230062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433368.358019:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433368.358029:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433368.358033:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433368.358042:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5737 00000100:00080000:3.0:1789433368.358050:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433368.358061:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433368.358107:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913719808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433368.358142:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433368.358146:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433368.358148:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433368.358152:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5737 00000100:00080000:3.0:1789433368.358156:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433368.358160:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433368.358172:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433368.358186:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913719936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433368.358196:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433368.358199:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433368.358201:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433368.358205:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5737 00000100:00080000:3.0:1789433368.358209:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433368.358215:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433368.358224:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913719808:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433368.358226:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433368.358247:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913720064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433368.358260:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433368.358263:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433368.358265:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433368.358270:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5737 00000100:00080000:3.0:1789433368.358273:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433368.358280:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433368.358284:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433368.358296:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913720192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433368.358306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433368.358371:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433368/real 1789433368] req@ffff9811d1bc6300 x1876350913719808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433368.358396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913719808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433368.358407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433368.358410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433368.358415:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913719808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433368.358423:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433368.358449:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433368.358462:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433368/real 1789433368] req@ffff9811d1bc4700 x1876350913719936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433368.358475:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913719936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433368.358480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433368.358483:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433368.358485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913719936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433368.358508:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433368.358528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433368.358540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433368/real 1789433368] req@ffff9811d1bc6680 x1876350913720064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433368.358551:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913720064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433368.358558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433368.358561:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433368.358563:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433368.358570:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720192:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433368.358589:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433368.358598:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433368/real 1789433368] req@ffff9811d1bc5500 x1876350913720192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433368.358606:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913720192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433368.358619:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433368.358621:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433368.358624:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433369.254051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433370.278379:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433371.301985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433372.326047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433373.350066:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433373.477969:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433373.477980:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433373.477984:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433373.477993:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5743 00000100:00080000:2.0:1789433373.478000:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433373.478011:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433373.478045:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913720320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433373.478087:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433373.478091:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433373.478093:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433373.478097:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5743 00000100:00080000:2.0:1789433373.478101:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433373.478106:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433373.478116:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433373.478129:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913720320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433373.478131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913720448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433373.478144:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433373.478147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433373.478150:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433373.478153:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5743 00000100:00080000:2.0:1789433373.478157:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433373.478161:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433373.478166:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433373.478177:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913720576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433373.478185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433373.478194:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433373.478197:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433373.478199:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433373.478204:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5743 00000100:00080000:2.0:1789433373.478207:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433373.478213:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433373.478218:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433373.478219:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433373/real 1789433373] req@ffff9811d2103b80 x1876350913720320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789433373.478235:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913720704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433373.478251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913720320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433373.478262:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433373.478267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433373.478272:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913720320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433373.478302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433373.478327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433373.478339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433373/real 1789433373] req@ffff9811d2101c00 x1876350913720448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433373.478350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913720448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433373.478370:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433373.478374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433373.478377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433373.478383:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433373.478401:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433373.478423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433373/real 1789433373] req@ffff9811d2102a00 x1876350913720576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433373.478434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913720576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433373.478441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433373.478444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433373.478447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433373.478452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720704:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433373.478470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433373.478481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433373/real 1789433373] req@ffff9811d2100e00 x1876350913720704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433373.478491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913720704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433373.478499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433373.478501:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433373.478505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720704:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433374.374013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433375.398021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433376.422000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433377.446008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433378.470062:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433378.597962:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433378.597969:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433378.597973:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433378.597980:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5748 00000100:00080000:0.0:1789433378.597987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433378.597995:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433378.598037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913720832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433378.598058:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433378.598061:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433378.598062:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433378.598065:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5748 00000100:00080000:0.0:1789433378.598068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433378.598071:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433378.598078:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433378.598086:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913720960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433378.598093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433378.598096:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433378.598098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789433378.598099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913720832:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433378.598103:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5748 00000100:00080000:0.0:1789433378.598107:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433378.598109:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433378.598112:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433378.598120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913721088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433378.598127:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433378.598129:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433378.598131:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433378.598133:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5748 00000100:00080000:0.0:1789433378.598136:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433378.598138:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433378.598141:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433378.598149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913721216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433378.598152:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433378.598183:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433378/real 1789433378] req@ffff9811c8cd6680 x1876350913720832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433378.598195:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913720832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433378.598202:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433378.598205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433378.598207:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913720832:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433378.598222:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433378.598235:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433378.598242:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433378/real 1789433378] req@ffff9811c8cd4700 x1876350913720960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433378.598249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913720960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433378.598254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433378.598256:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433378.598258:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913720960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433378.598261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433378.598272:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433378.598277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433378/real 1789433378] req@ffff9811c8cd5c00 x1876350913721088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433378.598283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913721088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433378.598288:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433378.598289:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433378.598291:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433378.598294:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721216:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433378.598304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433378.598312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433378/real 1789433378] req@ffff9811c8cd7100 x1876350913721216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433378.598323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913721216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433378.598330:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433378.598333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433378.598335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433379.494261:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433380.517991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433381.542015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433382.565915:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433383.590027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433383.717942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433383.717952:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433383.717956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433383.717966:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5753 00000100:00080000:3.0:1789433383.717985:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433383.717997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433383.718038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913721344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433383.718064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433383.718067:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433383.718069:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433383.718073:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5753 00000100:00080000:3.0:1789433383.718077:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433383.718083:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433383.718092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433383.718101:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913721344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433383.718108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913721472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433383.718119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433383.718122:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433383.718124:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433383.718128:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5753 00000100:00080000:3.0:1789433383.718131:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433383.718136:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433383.718140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433383.718151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913721600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433383.718161:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433383.718165:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433383.718167:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433383.718171:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433383.718176:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5753 00000100:00080000:3.0:1789433383.718180:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433383.718186:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433383.718191:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433383.718204:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913721728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433383.718210:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433383/real 1789433383] req@ffff9811d1bc7100 x1876350913721344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433383.718231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913721344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433383.718239:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433383.718242:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433383.718245:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913721344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433383.718262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433383.718283:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433383.718293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433383/real 1789433383] req@ffff9811d1bc5500 x1876350913721472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433383.718301:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913721472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433383.718305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433383.718308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433383.718311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433383.718316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433383.718333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433383.718344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433383/real 1789433383] req@ffff9811d1bc6680 x1876350913721600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433383.718355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913721600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433383.718362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433383.718365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433383.718368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433383.718374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721728:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433383.718393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433383.718405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433383/real 1789433383] req@ffff9811d1bc4700 x1876350913721728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433383.718415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913721728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433383.718423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433383.718426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433383.718428:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721728:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433384.614000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433385.637942:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433386.661966:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433387.686060:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433388.710052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433388.837999:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433388.838015:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433388.838020:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433388.838034:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5758 00000100:00080000:2.0:1789433388.838047:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433388.838061:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433388.838108:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913721856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433388.838140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433388.838144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433388.838147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433388.838151:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5758 00000100:00080000:2.0:1789433388.838155:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433388.838164:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433388.838176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433388.838194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913721984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433388.838196:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913721856:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433388.838218:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433388.838222:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433388.838224:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433388.838229:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5758 00000100:00080000:2.0:1789433388.838234:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433388.838239:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433388.838245:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433388.838259:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913722112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433388.838272:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433388.838278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433388.838281:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433388.838285:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433388.838290:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5758 00000100:00080000:2.0:1789433388.838296:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433388.838306:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433388.838311:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433388.838329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433388/real 1789433388] req@ffff9811d2101880 x1876350913721856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789433388.838331:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913722240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433388.838378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913721856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433388.838392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433388.838397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433388.838403:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913721856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433388.838430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433388.838454:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433388.838467:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433388/real 1789433388] req@ffff9811d2101180 x1876350913721984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433388.838479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913721984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433388.838494:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433388.838498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433388.838501:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913721984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433388.838507:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433388.838526:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433388.838536:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433388/real 1789433388] req@ffff9811d2101f80 x1876350913722112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433388.838555:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913722112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433388.838562:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433388.838565:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433388.838568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433388.838575:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722240:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433388.838594:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433388.838610:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433388/real 1789433388] req@ffff9811d2100e00 x1876350913722240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433388.838622:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913722240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433388.838630:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433388.838633:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433388.838636:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722240:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433389.734081:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433390.757959:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433391.782028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433392.806047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433393.830023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433393.957922:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433393.957932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433393.957936:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433393.957945:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5763 00000100:00080000:2.0:1789433393.957953:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433393.957965:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433393.957999:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913722368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433393.958025:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433393.958029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433393.958032:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433393.958036:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5763 00000100:00080000:2.0:1789433393.958039:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433393.958044:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433393.958055:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433393.958069:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913722368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433393.958082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913722496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433393.958094:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433393.958097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433393.958099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433393.958103:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5763 00000100:00080000:2.0:1789433393.958107:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433393.958112:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433393.958118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433393.958134:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913722624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433393.958145:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433393.958149:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433393.958152:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433393.958156:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5763 00000100:00100000:0.0:1789433393.958160:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722496:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789433393.958161:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433393.958172:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433393.958179:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433393.958190:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789433393.958194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913722752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433393.958232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722752:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433394.854070:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433395.877992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433396.902088:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433397.030280:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433397.030339:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433397.030358:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433397.030406:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433397.030416:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433397.030425:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433397.030518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433393/real 1789433397] req@ffff9811d2101500 x1876350913722752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433397.030547:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913722752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433397.030560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433397.030565:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433397.030569:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722752:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433397.030576:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433393/real 1789433397] req@ffff9811d2101c00 x1876350913722496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433397.030588:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913722496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433397.030596:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433397.030601:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433397.030604:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433397.030611:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433393/real 1789433397] req@ffff9811d2103b80 x1876350913722624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433397.030623:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913722624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433397.030631:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433397.030635:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433397.030638:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913722624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433397.030645:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433393/real 1789433397] req@ffff9811d2102a00 x1876350913722368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433397.030657:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913722368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433397.030663:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433397.030667:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433397.030670:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913722368:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789433397.926016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433398.950135:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433399.078006:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433399.078020:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433399.078025:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433399.078038:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5768 00000100:00080000:3.0:1789433399.078050:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433399.078063:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433399.078106:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913722880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433399.078297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913722880:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433399.078374:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433399.078453:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433399.078471:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433399.078474:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433399.078480:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5768 00000100:00080000:3.0:1789433399.078487:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789433399.078489:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433399/real 1789433399] req@ffff9811d1bc6300 x1876350913722880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433399.078501:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433399.078524:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433399.078525:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913722880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433399.078545:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789433399.078552:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913723008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433399.078556:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433399.078569:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913722880:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433399.078589:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433399.078594:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433399.078596:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433399.078605:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5768 00000100:00100000:0.0:1789433399.078610:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723008:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789433399.078613:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433399.078627:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433399.078636:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789433399.078651:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433399.078663:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433399/real 1789433399] req@ffff9811d1bc5880 x1876350913723008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433399.078676:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913723008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433399.078684:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433399.078687:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433399.078691:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433399.078736:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913723136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433399.078758:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433399.078761:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433399.078763:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433399.078767:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5768 00000100:00080000:3.0:1789433399.078771:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433399.078776:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433399.078783:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433399.078794:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913723264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433399.079204:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433399.079235:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433399.079248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433399/real 1789433399] req@ffff9811d1bc5f80 x1876350913723136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433399.079266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913723136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433399.079275:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433399.079278:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433399.079281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433399.079288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723264:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433399.079304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433399.079314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433399/real 1789433399] req@ffff9811d1bc4000 x1876350913723264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433399.079324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913723264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433399.079332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433399.079335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433399.079338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723264:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433399.974024:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433400.998011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433402.021996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433403.046036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433404.070299:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433404.197954:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433404.197968:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433404.197973:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433404.197988:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5773 00000100:00080000:3.0:1789433404.198000:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433404.198016:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433404.198064:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913723392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433404.198100:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433404.198104:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433404.198106:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433404.198111:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5773 00000100:00080000:3.0:1789433404.198115:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433404.198120:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433404.198132:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433404.198147:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913723520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433404.198158:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433404.198162:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433404.198164:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433404.198169:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5773 00000100:00080000:3.0:1789433404.198174:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433404.198179:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433404.198184:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433404.198193:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913723392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433404.198202:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913723648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433404.198215:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433404.198218:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433404.198219:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433404.198224:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5773 00000100:00080000:3.0:1789433404.198228:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433404.198232:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433404.198237:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433404.198252:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913723776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433404.198274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433404.198319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433404/real 1789433404] req@ffff9811d1bc5c00 x1876350913723392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433404.198343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913723392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433404.198353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433404.198357:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433404.198361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913723392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433404.198369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433404.198392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433404.198404:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433404/real 1789433404] req@ffff9811d1bc7800 x1876350913723520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433404.198416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913723520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433404.198423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433404.198426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433404.198429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433404.198453:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433404.198472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433404.198483:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433404/real 1789433404] req@ffff9811d1bc5180 x1876350913723648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433404.198494:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913723648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433404.198506:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433404.198509:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433404.198513:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433404.198518:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723776:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433404.198537:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433404.198547:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433404/real 1789433404] req@ffff9811d1bc6a00 x1876350913723776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433404.198558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913723776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433404.198566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433404.198569:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433404.198572:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913723776:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433405.094098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433406.118117:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433407.142109:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433408.166980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433409.190186:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433409.317990:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433409.317999:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433409.318004:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433409.318014:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5778 00000100:00080000:2.0:1789433409.318022:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433409.318030:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433409.318060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913723904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433409.318086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433409.318089:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433409.318090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433409.318093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5778 00000100:00080000:2.0:1789433409.318095:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433409.318097:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433409.318105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433409.318113:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913724032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433409.318119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433409.318121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433409.318122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433409.318124:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5778 00000100:00080000:2.0:1789433409.318126:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433409.318128:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433409.318131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433409.318137:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913724160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433409.318143:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433409.318145:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433409.318146:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433409.318148:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5778 00000100:00080000:2.0:1789433409.318150:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433409.318152:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433409.318154:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433409.318160:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913724288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433409.318870:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913723904:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433409.318938:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433409.318995:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433409/real 1789433409] req@ffff9811d2102d80 x1876350913723904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433409.319011:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913723904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433409.319026:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433409.319030:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433409.319034:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913723904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433409.319041:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433409.319062:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433409.319078:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433409/real 1789433409] req@ffff9811d2103800 x1876350913724032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433409.319089:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913724032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433409.319097:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433409.319100:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433409.319103:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433409.319108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433409.319123:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433409.319131:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433409/real 1789433409] req@ffff9811d2100000 x1876350913724160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433409.319140:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913724160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433409.319146:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433409.319149:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433409.319151:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433409.319156:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724288:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433409.319170:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433409.319189:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433409/real 1789433409] req@ffff9811d2102a00 x1876350913724288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433409.319199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913724288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433409.319206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433409.319209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433409.319211:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724288:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433410.213992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433411.238068:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433412.261995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433413.286056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433414.309990:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433414.437946:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433414.437956:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433414.437960:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433414.437970:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5784 00000100:00080000:3.0:1789433414.437979:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433414.437990:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433414.438024:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913724416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433414.438053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433414.438059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433414.438061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433414.438064:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5784 00000100:00080000:3.0:1789433414.438068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433414.438073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433414.438081:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433414.438097:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913724544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433414.438108:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433414.438111:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433414.438113:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433414.438117:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5784 00000100:00080000:3.0:1789433414.438121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433414.438126:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433414.438132:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433414.438136:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913724416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433414.438143:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913724672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433414.438153:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433414.438155:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433414.438164:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433414.438167:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5784 00000100:00080000:3.0:1789433414.438172:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433414.438176:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433414.438181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433414.438191:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913724800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433414.438225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433414.438281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433414/real 1789433414] req@ffff9811d1bc7480 x1876350913724416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433414.438310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913724416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433414.438322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433414.438326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433414.438328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913724416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433414.438338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433414.438362:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433414.438375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433414/real 1789433414] req@ffff9811d1bc6a00 x1876350913724544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433414.438387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913724544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433414.438394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433414.438397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433414.438399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433414.438422:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433414.438437:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433414.438443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433414/real 1789433414] req@ffff9811d1bc5180 x1876350913724672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433414.438450:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913724672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433414.438455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433414.438457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433414.438460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433414.438466:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724800:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433414.438484:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433414.438494:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433414/real 1789433414] req@ffff9811d1bc7800 x1876350913724800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433414.438512:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913724800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433414.438519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433414.438521:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433414.438523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913724800:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433415.334011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433416.358021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433417.382189:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433418.406025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433419.430011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433419.557945:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433419.557954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433419.557958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433419.557974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5789 00000100:00080000:0.0:1789433419.557983:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433419.557991:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433419.558020:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913724928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433419.558041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433419.558044:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433419.558045:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433419.558048:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5789 00000100:00080000:0.0:1789433419.558050:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433419.558052:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433419.558060:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433419.558068:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913725056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433419.558075:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433419.558077:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433419.558079:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433419.558083:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5789 00000100:00100000:3.0:1789433419.558083:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913724928:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433419.558087:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433419.558093:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433419.558096:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433419.558102:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913725184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433419.558108:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433419.558111:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433419.558112:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433419.558114:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5789 00000100:00080000:0.0:1789433419.558116:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433419.558119:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433419.558121:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433419.558127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913725312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433419.558143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433419.558175:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433419/real 1789433419] req@ffff9811c8cd7480 x1876350913724928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433419.558190:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913724928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433419.558200:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433419.558203:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433419.558207:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913724928:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433419.558224:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433419.558242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433419.558252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433419/real 1789433419] req@ffff9811c8cd7800 x1876350913725056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433419.558262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913725056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433419.558269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433419.558272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433419.558275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433419.558280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433419.558295:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433419.558303:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433419/real 1789433419] req@ffff9811c8cd5180 x1876350913725184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433419.558313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913725184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433419.558319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433419.558322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433419.558324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433419.558330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725312:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433419.558344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433419.558351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433419/real 1789433419] req@ffff9811c8cd4000 x1876350913725312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433419.558361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913725312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433419.558368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433419.558371:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433419.558380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725312:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433420.453998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433421.477976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433422.502004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433423.526114:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433424.550048:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433424.677951:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433424.677961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433424.677976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433424.677987:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5794 00000100:00080000:0.0:1789433424.677996:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433424.678006:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433424.678039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913725440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433424.678070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433424.678077:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433424.678079:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433424.678083:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5794 00000100:00080000:0.0:1789433424.678087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433424.678093:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433424.678101:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433424.678112:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913725568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433424.678113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913725440:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433424.678126:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433424.678128:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433424.678130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433424.678132:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5794 00000100:00080000:0.0:1789433424.678134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433424.678137:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433424.678140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433424.678148:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913725696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433424.678156:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433424.678158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433424.678159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433424.678161:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5794 00000100:00080000:0.0:1789433424.678163:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433424.678165:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433424.678168:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433424.678174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913725824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433424.678180:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433424.678221:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433424/real 1789433424] req@ffff9811c8cd6300 x1876350913725440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433424.678244:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913725440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433424.678254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433424.678258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433424.678262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913725440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433424.678282:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433424.678304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433424.678315:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433424/real 1789433424] req@ffff9811c8cd7b80 x1876350913725568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433424.678327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913725568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433424.678335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433424.678338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433424.678342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433424.678347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433424.678367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433424.678377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433424/real 1789433424] req@ffff9811c8cd6a00 x1876350913725696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433424.678388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913725696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433424.678401:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433424.678404:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433424.678407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433424.678413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725824:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433424.678430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433424.678439:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433424/real 1789433424] req@ffff9811c8cd4a80 x1876350913725824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433424.678449:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913725824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433424.678457:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433424.678460:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433424.678463:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913725824:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433425.574018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433426.598006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433427.621976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433428.646018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433429.669974:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433429.798014:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433429.798030:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433429.798034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433429.798044:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5799 00000100:00080000:0.0:1789433429.798053:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433429.798064:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433429.798104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913725952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433429.798128:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433429.798130:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433429.798132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433429.798137:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5799 00000100:00080000:0.0:1789433429.798140:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433429.798145:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433429.798157:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433429.798175:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913726080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433429.798182:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913725952:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433429.798195:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433429.798200:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433429.798203:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433429.798215:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5799 00000100:00080000:0.0:1789433429.798220:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433429.798226:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433429.798234:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433429.798245:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913726208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433429.798257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433429.798261:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433429.798266:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433429.798267:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433429.798271:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5799 00000100:00080000:0.0:1789433429.798277:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433429.798283:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433429.798290:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433429.798299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433429/real 1789433429] req@ffff9811c8cd5500 x1876350913725952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433429.798306:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913726336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433429.798353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913725952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433429.798366:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433429.798370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433429.798374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913725952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433429.798399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433429.798432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433429.798446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433429/real 1789433429] req@ffff9811c8cd6d80 x1876350913726080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433429.798458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913726080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433429.798466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433429.798469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433429.798473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433429.798479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433429.798496:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433429.798507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433429/real 1789433429] req@ffff9811c8cd5f80 x1876350913726208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433429.798518:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913726208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433429.798525:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433429.798529:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433429.798532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433429.798539:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726336:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433429.798557:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433429.798567:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433429/real 1789433429] req@ffff9811c8cd4a80 x1876350913726336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433429.798578:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913726336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433429.798586:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433429.798589:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433429.798592:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726336:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433430.693955:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433431.717950:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433432.742056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433433.766051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433434.790040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433434.917954:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433434.917973:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433434.917982:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433434.917993:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5804 00000100:00080000:0.0:1789433434.918002:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433434.918014:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433434.918056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913726464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433434.918082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433434.918085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433434.918086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433434.918089:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5804 00000100:00080000:0.0:1789433434.918092:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433434.918095:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433434.918104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433434.918114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913726592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433434.918122:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433434.918126:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433434.918130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789433434.918131:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913726464:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433434.918136:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5804 00000100:00080000:0.0:1789433434.918148:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433434.918154:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433434.918159:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433434.918171:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913726720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433434.918180:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433434.918183:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433434.918184:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433434.918187:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5804 00000100:00080000:0.0:1789433434.918192:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433434.918196:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433434.918202:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433434.918217:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913726848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433434.918218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433434.918275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433434/real 1789433434] req@ffff9811c8cd6a00 x1876350913726464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433434.918291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913726464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433434.918300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433434.918304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433434.918308:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913726464:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433434.918335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433434.918357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433434.918367:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433434/real 1789433434] req@ffff9811c8cd7b80 x1876350913726592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433434.918375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913726592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433434.918389:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433434.918393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433434.918396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433434.918402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433434.918421:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433434.918430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433434/real 1789433434] req@ffff9811c8cd6300 x1876350913726720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433434.918437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913726720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433434.918442:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433434.918445:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433434.918447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433434.918451:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726848:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433434.918467:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433434.918477:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433434/real 1789433434] req@ffff9811c8cd4000 x1876350913726848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433434.918489:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913726848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433434.918497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433434.918500:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433434.918504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913726848:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433435.814036:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433436.838008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433437.861977:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433438.885987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433439.910009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433440.037941:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433440.037961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433440.037965:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433440.037975:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5809 00000100:00080000:0.0:1789433440.037983:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433440.037992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433440.038021:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913726976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433440.038040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433440.038042:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433440.038044:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433440.038046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5809 00000100:00080000:0.0:1789433440.038048:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433440.038050:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433440.038058:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433440.038076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913727104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433440.038076:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913726976:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433440.038091:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433440.038093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433440.038094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433440.038097:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5809 00000100:00080000:0.0:1789433440.038099:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433440.038101:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433440.038108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433440.038115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913727232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433440.038121:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433440.038124:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433440.038125:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433440.038128:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5809 00000100:00080000:0.0:1789433440.038130:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433440.038133:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433440.038137:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789433440.038142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433440.038146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913727360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433440.038196:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433440/real 1789433440] req@ffff9811c8cd5180 x1876350913726976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433440.038215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913726976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433440.038225:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433440.038229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433440.038233:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913726976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433440.038261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433440.038282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433440.038292:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433440/real 1789433440] req@ffff9811c8cd7800 x1876350913727104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433440.038303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913727104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433440.038312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433440.038315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433440.038318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433440.038323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433440.038350:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433440.038361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433440/real 1789433440] req@ffff9811c8cd7480 x1876350913727232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433440.038372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913727232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433440.038379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433440.038382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433440.038385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433440.038391:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727360:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433440.038408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433440.038419:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433440/real 1789433440] req@ffff9811c8cd7100 x1876350913727360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433440.038430:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913727360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433440.038438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433440.038441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433440.038444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727360:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433440.934061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433441.958003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433442.982013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433444.006033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433445.030000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433445.158024:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433445.158033:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433445.158037:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433445.158046:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5814 00000100:00080000:3.0:1789433445.158054:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433445.158064:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433445.158099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913727488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433445.158159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913727488:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433445.158218:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433445.158248:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433445.158251:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433445.158253:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433445.158257:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5814 00000100:00080000:3.0:1789433445.158261:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433445.158267:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433445.158275:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433445.158288:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913727616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433445.158353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433445/real 1789433445] req@ffff9811d1bc5c00 x1876350913727488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433445.158370:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913727488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433445.158379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433445.158383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433445.158387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913727488:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433445.158408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433445.158429:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433445.158441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433445/real 1789433445] req@ffff9811d1bc4000 x1876350913727616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433445.158452:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913727616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433445.158459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433445.158462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433445.158466:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727616:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789433445.158509:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433445.158513:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433445.158515:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433445.158519:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5814 00000100:00080000:3.0:1789433445.158534:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433445.158539:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433445.158544:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433445.158556:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913727744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433445.158567:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433445.158579:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433445.158581:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433445.158585:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5814 00000100:00080000:3.0:1789433445.158588:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433445.158592:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433445.158597:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433445.158607:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913727872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433445.158664:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433445.158686:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433445.158697:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433445/real 1789433445] req@ffff9811d1bc5f80 x1876350913727744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433445.158709:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913727744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433445.158717:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433445.158719:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433445.158722:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433445.158729:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727872:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433445.158757:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433445.158768:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433445/real 1789433445] req@ffff9811d1bc5880 x1876350913727872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433445.158779:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913727872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433445.158786:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433445.158788:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433445.158791:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913727872:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433446.054055:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433447.078085:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433448.102030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433449.125980:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433450.149991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433450.277922:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433450.277931:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433450.277934:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433450.277944:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5819 00000100:00080000:2.0:1789433450.277952:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433450.277962:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433450.277994:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913728000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433450.278016:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433450.278019:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433450.278021:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433450.278025:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5819 00000100:00080000:2.0:1789433450.278028:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433450.278032:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433450.278040:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433450.278051:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913728128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433450.278066:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433450.278067:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913728000:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433450.278073:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433450.278077:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433450.278081:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5819 00000100:00080000:2.0:1789433450.278084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433450.278088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433450.278092:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433450.278102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913728256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433450.278112:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433450.278115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433450.278117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433450.278121:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5819 00000100:00080000:2.0:1789433450.278124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433450.278128:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433450.278133:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789433450.278136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433450.278150:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913728384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433450.278183:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433450/real 1789433450] req@ffff9811d2103b80 x1876350913728000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433450.278203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913728000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433450.278213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433450.278217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433450.278221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913728000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433450.278241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433450.278264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433450.278276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433450/real 1789433450] req@ffff9811d2101c00 x1876350913728128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433450.278288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913728128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433450.278298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433450.278301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433450.278304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433450.278310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433450.278326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433450.278336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433450/real 1789433450] req@ffff9811d2101500 x1876350913728256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433450.278352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913728256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433450.278359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433450.278362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433450.278365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433450.278370:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728384:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433450.278387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433450.278396:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433450/real 1789433450] req@ffff9811d2100e00 x1876350913728384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433450.278406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913728384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433450.278414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433450.278416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433450.278419:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728384:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433451.173980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433452.198061:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433453.221997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433454.245989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433455.269997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433455.397926:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433455.397934:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433455.397937:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433455.397946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5824 00000100:00080000:2.0:1789433455.397952:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433455.397961:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433455.397988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913728512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433455.398010:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433455.398013:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433455.398014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433455.398017:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5824 00000100:00080000:2.0:1789433455.398018:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433455.398021:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433455.398028:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433455.398036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913728640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433455.398042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433455.398045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433455.398046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433455.398048:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5824 00000100:00080000:2.0:1789433455.398050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433455.398053:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433455.398055:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433455.398060:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913728512:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433455.398064:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913728768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433455.398076:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433455.398079:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433455.398082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433455.398085:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5824 00000100:00080000:2.0:1789433455.398089:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433455.398094:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433455.398100:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433455.398112:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913728896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433455.398122:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433455.398168:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433455/real 1789433455] req@ffff9811d2101f80 x1876350913728512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433455.398189:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913728512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433455.398200:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433455.398205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433455.398208:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913728512:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433455.398216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433455.398237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433455.398249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433455/real 1789433455] req@ffff9811d2101180 x1876350913728640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433455.398261:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913728640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433455.398269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433455.398272:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433455.398276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433455.398296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433455.398317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433455.398329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433455/real 1789433455] req@ffff9811d2101880 x1876350913728768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433455.398340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913728768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433455.398348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433455.398351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433455.398355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433455.398361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728896:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433455.398381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433455.398391:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433455/real 1789433455] req@ffff9811d2100e00 x1876350913728896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433455.398403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913728896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433455.398411:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433455.398414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433455.398417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913728896:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433456.294011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433457.318013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433458.341965:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433459.365997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433460.389996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433460.517947:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433460.517956:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433460.517960:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433460.517969:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5830 00000100:00080000:3.0:1789433460.517976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433460.517987:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433460.518021:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913729024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433460.518047:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433460.518051:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433460.518052:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433460.518057:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5830 00000100:00080000:3.0:1789433460.518060:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433460.518065:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433460.518074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433460.518086:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913729152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433460.518089:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913729024:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433460.518102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433460.518105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433460.518107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433460.518110:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5830 00000100:00080000:3.0:1789433460.518112:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433460.518115:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433460.518118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433460.518127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913729280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433460.518134:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433460.518136:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433460.518137:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433460.518140:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5830 00000100:00080000:3.0:1789433460.518142:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433460.518146:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433460.518150:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433460.518161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913729408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433460.518189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433460.518218:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433460.518232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729408:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433461.414022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433462.437998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433463.462045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789433463.590256:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789433463.590303:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789433463.590318:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433463.590366:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433463.590376:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433463.590391:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433463.590574:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433460/real 1789433463] req@ffff9811d1bc4700 x1876350913729152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433463.590596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913729152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433463.590608:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433463.590614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433463.590618:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433463.590625:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433460/real 1789433463] req@ffff9811d1bc6680 x1876350913729280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433463.590638:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913729280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433463.590651:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433463.590654:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433463.590657:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433463.590665:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433460/real 1789433463] req@ffff9811d1bc5500 x1876350913729408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433463.590677:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913729408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433463.590684:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433463.590688:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433463.590691:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729408:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433463.590697:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433460/real 1789433463] req@ffff9811d1bc6300 x1876350913729024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433463.590708:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913729024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433463.590714:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433463.590718:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433463.590720:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913729024:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789433464.485966:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433465.510190:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433465.638006:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433465.638016:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433465.638020:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433465.638031:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5835 00000100:00080000:1.0:1789433465.638038:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433465.638050:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433465.638080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913729536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433465.638105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433465.638109:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433465.638111:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433465.638114:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5835 00000100:00080000:1.0:1789433465.638118:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433465.638123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433465.638133:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433465.638146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913729664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433465.638157:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913729536:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433465.638158:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433465.638167:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433465.638169:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433465.638173:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5835 00000100:00080000:1.0:1789433465.638177:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433465.638181:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433465.638187:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433465.638199:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913729792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433465.638220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433465.638220:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433465.638227:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433465.638229:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433465.638234:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5835 00000100:00080000:1.0:1789433465.638238:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433465.638242:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433465.638247:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433465.638260:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913729920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433465.638279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433465/real 1789433465] req@ffff9811d0bddc00 x1876350913729536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433465.638299:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913729536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433465.638308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433465.638311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433465.638315:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913729536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433465.638348:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433465.638386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433465.638398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433465/real 1789433465] req@ffff9811d0bdc380 x1876350913729664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433465.638410:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913729664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433465.638417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433465.638420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433465.638423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433465.638429:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433465.638446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433465.638455:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433465/real 1789433465] req@ffff9811d0bdd180 x1876350913729792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433465.638467:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913729792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433465.638474:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433465.638477:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433465.638480:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433465.638486:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433465.638503:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433465.638512:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433465/real 1789433465] req@ffff9811d0bdd500 x1876350913729920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433465.638522:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913729920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433465.638531:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433465.638533:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433465.638536:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913729920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433466.533993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433467.558020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433468.581961:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433469.605967:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433470.630002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433470.757951:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433470.757962:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433470.757966:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433470.757976:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5840 00000100:00080000:3.0:1789433470.757984:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433470.757995:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433470.758022:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913730048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433470.758044:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433470.758046:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433470.758050:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433470.758053:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5840 00000100:00080000:3.0:1789433470.758055:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433470.758058:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433470.758065:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433470.758074:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913730176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433470.758082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433470.758085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433470.758087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433470.758088:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913730048:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433470.758091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5840 00000100:00080000:3.0:1789433470.758097:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433470.758099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433470.758102:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433470.758109:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913730304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433470.758119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433470.758122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433470.758123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433470.758126:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5840 00000100:00080000:3.0:1789433470.758128:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433470.758131:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433470.758134:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433470.758141:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913730432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433470.758148:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433470.758192:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433470/real 1789433470] req@ffff9811d1bc7100 x1876350913730048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433470.758217:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913730048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433470.758228:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433470.758233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433470.758237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913730048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433470.758258:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433470.758281:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433470.758293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433470/real 1789433470] req@ffff9811d1bc6300 x1876350913730176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433470.758302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913730176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433470.758307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433470.758309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433470.758311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433470.758315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433470.758327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433470.758349:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433470/real 1789433470] req@ffff9811d1bc5500 x1876350913730304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433470.758357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913730304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433470.758365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433470.758367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433470.758369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433470.758373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433470.758386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433470.758392:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433470/real 1789433470] req@ffff9811d1bc6680 x1876350913730432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433470.758398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913730432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433470.758403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433470.758405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433470.758407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433471.653985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433472.678015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433473.701984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433474.726000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433475.750003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433475.877929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433475.877937:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433475.877940:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433475.877948:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5845 00000100:00080000:1.0:1789433475.877956:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433475.877965:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433475.877994:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913730560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433475.878012:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433475.878014:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433475.878015:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433475.878018:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5845 00000100:00080000:1.0:1789433475.878020:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433475.878022:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433475.878030:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433475.878037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913730688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433475.878045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433475.878049:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789433475.878051:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913730560:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433475.878051:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433475.878055:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5845 00000100:00080000:1.0:1789433475.878060:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433475.878063:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433475.878065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433475.878071:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913730816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433475.878078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433475.878080:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433475.878081:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433475.878083:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5845 00000100:00080000:1.0:1789433475.878085:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433475.878087:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433475.878090:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433475.878096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913730944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433475.878119:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433475.878165:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433475/real 1789433475] req@ffff9811d0bde680 x1876350913730560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433475.878183:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913730560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433475.878192:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433475.878196:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433475.878200:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913730560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433475.878220:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433475.878242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433475.878254:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433475/real 1789433475] req@ffff9811d0bdf480 x1876350913730688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433475.878266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913730688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433475.878273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433475.878276:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433475.878279:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433475.878285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433475.878320:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433475.878344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433475/real 1789433475] req@ffff9811d0bdfb80 x1876350913730816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433475.878357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913730816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433475.878365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433475.878368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433475.878371:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433475.878377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730944:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433475.878396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433475.878407:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433475/real 1789433475] req@ffff9811d0bdc700 x1876350913730944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433475.878419:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913730944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433475.878427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433475.878430:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433475.878434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913730944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433476.774027:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433477.797946:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433478.822045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433479.846034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433480.870000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433480.997967:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433480.997978:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433480.997982:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433480.997992:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5850 00000100:00080000:3.0:1789433480.998001:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433480.998010:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433480.998038:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913731072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433480.998059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433480.998066:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433480.998068:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433480.998073:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5850 00000100:00080000:3.0:1789433480.998075:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433480.998078:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433480.998086:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433480.998096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913731200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433480.998106:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913731072:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433480.998109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433480.998113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433480.998115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433480.998119:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5850 00000100:00080000:3.0:1789433480.998123:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433480.998129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433480.998135:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433480.998154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913731328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433480.998162:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433480.998164:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433480.998165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433480.998168:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433480.998172:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5850 00000100:00080000:3.0:1789433480.998176:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433480.998180:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433480.998184:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433480.998194:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913731456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433480.998214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433480/real 1789433480] req@ffff9811d1bc4700 x1876350913731072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433480.998227:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913731072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433480.998234:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433480.998236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433480.998239:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913731072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433480.998255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433480.998288:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433480.998301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433480/real 1789433480] req@ffff9811d1bc5880 x1876350913731200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433480.998315:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913731200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433480.998323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433480.998326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433480.998329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433480.998335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433480.998354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433480.998365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433480/real 1789433480] req@ffff9811d1bc5f80 x1876350913731328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433480.998377:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913731328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433480.998385:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433480.998388:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433480.998391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433480.998397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433480.998414:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433480.998423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433480/real 1789433480] req@ffff9811d1bc4000 x1876350913731456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433480.998434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913731456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433480.998448:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433480.998452:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433480.998455:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433481.894008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433482.918001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433483.942068:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433484.966020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433485.990407:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433486.117933:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433486.117944:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433486.117949:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433486.117959:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5855 00000100:00080000:3.0:1789433486.117967:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433486.117978:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433486.118010:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913731584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433486.118035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433486.118039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433486.118041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433486.118044:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5855 00000100:00080000:3.0:1789433486.118048:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433486.118052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433486.118061:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433486.118073:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913731712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433486.118085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433486.118089:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913731584:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433486.118089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433486.118092:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433486.118099:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5855 00000100:00080000:3.0:1789433486.118102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433486.118105:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433486.118109:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433486.118120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913731840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433486.118145:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433486.118148:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433486.118150:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433486.118153:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433486.118157:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5855 00000100:00080000:3.0:1789433486.118161:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433486.118168:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433486.118172:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433486.118185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913731968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433486.118195:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433486/real 1789433486] req@ffff9811d1bc5c00 x1876350913731584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433486.118215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913731584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433486.118237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433486.118239:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433486.118242:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913731584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433486.118259:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433486.118276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433486.118284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433486/real 1789433486] req@ffff9811d1bc7800 x1876350913731712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433486.118292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913731712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433486.118297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433486.118298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433486.118301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433486.118304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433486.118314:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433486.118319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433486/real 1789433486] req@ffff9811d1bc5180 x1876350913731840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433486.118325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913731840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433486.118330:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433486.118331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433486.118333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433486.118336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731968:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433486.118345:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433486.118350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433486/real 1789433486] req@ffff9811d1bc6a00 x1876350913731968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433486.118356:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913731968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433486.118361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433486.118362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433486.118364:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913731968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433487.013998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433488.037943:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433489.062001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433490.085950:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433491.110016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433491.237916:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433491.237925:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433491.237929:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433491.237938:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5860 00000100:00080000:0.0:1789433491.237946:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433491.237956:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433491.237988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913732096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433491.238014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433491.238018:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433491.238020:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433491.238032:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5860 00000100:00080000:0.0:1789433491.238037:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433491.238039:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913732096:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433491.238045:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433491.238054:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433491.238070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913732224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433491.238081:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433491.238084:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433491.238086:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433491.238089:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5860 00000100:00080000:0.0:1789433491.238093:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789433491.238095:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433491.238100:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433491.238107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433491.238119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913732352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433491.238130:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433491.238134:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1789433491.238134:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433491/real 1789433491] req@ffff9811c8cd5c00 x1876350913732096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433491.238138:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433491.238147:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5860 00000100:00080000:0.0:1789433491.238153:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433491.238158:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433491.238163:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913732096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789433491.238165:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789433491.238177:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789433491.238196:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913732480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433491.238196:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433491.238202:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913732096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433491.238230:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433491.238254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433491.238267:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433491/real 1789433491] req@ffff9811c8cd4700 x1876350913732224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433491.238285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913732224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433491.238290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433491.238292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433491.238294:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433491.238298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433491.238310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433491.238316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433491/real 1789433491] req@ffff9811c8cd6680 x1876350913732352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433491.238323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913732352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433491.238328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433491.238330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433491.238331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433491.238335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433491.238344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433491.238348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433491/real 1789433491] req@ffff9811c8cd7100 x1876350913732480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433491.238354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913732480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433491.238363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433491.238365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433491.238367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433492.133989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433493.157979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433494.181986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433495.205985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433496.229986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433496.357952:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433496.357962:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433496.357966:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433496.357976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5865 00000100:00080000:3.0:1789433496.357984:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433496.358000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433496.358032:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913732608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433496.358059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433496.358063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433496.358064:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433496.358067:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5865 00000100:00080000:3.0:1789433496.358070:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433496.358074:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433496.358081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433496.358092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913732736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433496.358100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913732608:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433496.358104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433496.358109:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433496.358111:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433496.358115:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5865 00000100:00080000:3.0:1789433496.358118:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433496.358123:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433496.358128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433496.358140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913732864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433496.358163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433496.358168:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433496.358168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433496.358171:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433496.358178:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5865 00000100:00080000:3.0:1789433496.358183:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433496.358190:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433496.358196:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433496.358206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433496/real 1789433496] req@ffff9811d1bc7480 x1876350913732608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433496.358212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913732992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433496.358232:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913732608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433496.358242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433496.358246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433496.358250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913732608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433496.358268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433496.358287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433496.358297:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433496/real 1789433496] req@ffff9811d1bc6a00 x1876350913732736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433496.358307:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913732736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433496.358314:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433496.358317:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433496.358319:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433496.358324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433496.358337:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433496.358344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433496/real 1789433496] req@ffff9811d1bc5180 x1876350913732864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433496.358353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913732864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433496.358359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433496.358362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433496.358365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433496.358369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433496.358384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433496.358392:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433496/real 1789433496] req@ffff9811d1bc7800 x1876350913732992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433496.358406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913732992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433496.358413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433496.358416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433496.358418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913732992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433497.254006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433498.278007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433499.302072:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433500.326026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433501.350019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433501.477957:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433501.477964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433501.477968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433501.477975:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5871 00000100:00080000:0.0:1789433501.477982:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433501.477990:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433501.478021:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913733120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433501.478040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433501.478042:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433501.478043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433501.478045:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5871 00000100:00080000:0.0:1789433501.478047:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433501.478050:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433501.478057:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433501.478066:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913733248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433501.478073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913733120:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433501.478075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433501.478080:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433501.478082:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433501.478085:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5871 00000100:00080000:0.0:1789433501.478086:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433501.478088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433501.478091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433501.478099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913733376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433501.478115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433501.478117:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433501.478119:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433501.478122:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5871 00000100:00080000:0.0:1789433501.478125:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433501.478129:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433501.478132:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433501.478140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913733504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433501.478149:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433501.478185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433501/real 1789433501] req@ffff9811c8cd7480 x1876350913733120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433501.478198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913733120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433501.478206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433501.478209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433501.478212:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913733120:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433501.478233:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433501.478254:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433501.478268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433501/real 1789433501] req@ffff9811c8cd7800 x1876350913733248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433501.478280:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913733248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433501.478287:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433501.478290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433501.478293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433501.478299:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433501.478316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433501.478327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433501/real 1789433501] req@ffff9811c8cd5180 x1876350913733376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433501.478339:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913733376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433501.478346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433501.478348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433501.478351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433501.478355:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733504:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433501.478370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433501.478376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433501/real 1789433501] req@ffff9811c8cd4000 x1876350913733504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433501.478383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913733504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433501.478388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433501.478390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433501.478392:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433502.374011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433503.397998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433504.422062:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433505.445988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433506.470078:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433506.597972:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433506.597982:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433506.597987:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433506.597997:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5876 00000100:00080000:0.0:1789433506.598006:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433506.598014:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433506.598049:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913733632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433506.598092:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433506.598096:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433506.598098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433506.598103:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5876 00000100:00080000:0.0:1789433506.598106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433506.598111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433506.598120:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433506.598132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913733760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433506.598142:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433506.598145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433506.598146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433506.598150:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5876 00000100:00080000:0.0:1789433506.598153:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433506.598157:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433506.598162:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433506.598173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913733888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433506.598189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433506.598193:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433506.598194:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433506.598197:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5876 00000100:00080000:0.0:1789433506.598199:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433506.598202:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433506.598205:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433506.598213:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913734016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433506.598312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913733632:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433506.598371:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433506.598413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433506/real 1789433506] req@ffff9811c8cd6300 x1876350913733632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433506.598430:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913733632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433506.598440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433506.598444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433506.598446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913733632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433506.598453:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433506.598472:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433506.598484:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433506/real 1789433506] req@ffff9811c8cd7b80 x1876350913733760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433506.598498:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913733760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433506.598503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433506.598506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433506.598509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433506.598515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433506.598533:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433506.598547:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433506/real 1789433506] req@ffff9811c8cd6a00 x1876350913733888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433506.598555:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913733888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433506.598559:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433506.598562:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433506.598564:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913733888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433506.598568:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433506.598586:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433506.598596:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433506/real 1789433506] req@ffff9811c8cd4a80 x1876350913734016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433506.598607:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913734016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433506.598619:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433506.598622:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433506.598624:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433507.493970:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433508.517996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433509.542001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433510.566051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433511.589998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433511.717932:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433511.717940:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433511.717943:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433511.717951:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5881 00000100:00080000:0.0:1789433511.717957:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433511.717966:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433511.717993:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913734144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433511.718014:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433511.718017:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433511.718019:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433511.718023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5881 00000100:00080000:0.0:1789433511.718026:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433511.718039:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433511.718048:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433511.718061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913734272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433511.718072:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433511.718076:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789433511.718076:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913734144:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433511.718078:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433511.718084:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5881 00000100:00080000:0.0:1789433511.718087:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433511.718090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433511.718093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433511.718102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913734400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433511.718108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433511.718111:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433511.718112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433511.718114:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5881 00000100:00080000:0.0:1789433511.718116:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433511.718118:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433511.718121:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433511.718127:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913734528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433511.718139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433511.718176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433511/real 1789433511] req@ffff9811c8cd5f80 x1876350913734144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433511.718192:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913734144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433511.718202:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433511.718206:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433511.718209:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913734144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433511.718228:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433511.718247:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433511.718258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433511/real 1789433511] req@ffff9811c8cd6d80 x1876350913734272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433511.718270:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913734272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433511.718278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433511.718281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433511.718284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433511.718290:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433511.718306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433511.718316:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433511/real 1789433511] req@ffff9811c8cd5500 x1876350913734400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433511.718326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913734400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433511.718332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433511.718335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433511.718337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433511.718342:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734528:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433511.718357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433511.718366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433511/real 1789433511] req@ffff9811c8cd4a80 x1876350913734528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433511.718375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913734528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433511.718381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433511.718384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433511.718387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433512.614018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433513.638059:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433514.662029:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433515.685950:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433516.709942:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433516.837936:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433516.837944:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433516.837949:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433516.837959:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5886 00000100:00080000:3.0:1789433516.837978:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433516.837989:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433516.838029:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913734656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433516.838050:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433516.838052:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433516.838054:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433516.838056:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5886 00000100:00080000:3.0:1789433516.838059:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433516.838064:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433516.838071:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433516.838080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913734656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433516.838081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913734784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433516.838093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433516.838095:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433516.838096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433516.838098:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5886 00000100:00080000:3.0:1789433516.838101:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433516.838103:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433516.838106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433516.838114:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913734912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433516.838124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433516.838126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433516.838127:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433516.838129:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5886 00000800:00000100:0.0:1789433516.838133:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433516.838134:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433516.838140:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433516.838145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433516.838158:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913735040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433516.838166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433516/real 1789433516] req@ffff9811d1bc5c00 x1876350913734656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433516.838179:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913734656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433516.838195:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433516.838198:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433516.838201:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913734656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433516.838217:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433516.838231:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433516.838238:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433516/real 1789433516] req@ffff9811d1bc4000 x1876350913734784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433516.838245:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913734784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433516.838250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433516.838252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433516.838254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433516.838257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433516.838267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433516.838272:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433516/real 1789433516] req@ffff9811d1bc5f80 x1876350913734912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433516.838279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913734912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433516.838283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433516.838284:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433516.838286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913734912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433516.838289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433516.838303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433516.838308:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433516/real 1789433516] req@ffff9811d1bc5880 x1876350913735040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433516.838315:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913735040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433516.838321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433516.838324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433516.838327:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433517.734018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433518.758004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433519.782030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433520.806025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433521.830009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433521.957915:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433521.957923:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433521.957926:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433521.957933:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5891 00000100:00080000:1.0:1789433521.957941:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433521.957949:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433521.957990:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913735168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433521.958010:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433521.958013:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433521.958015:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433521.958019:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5891 00000100:00080000:1.0:1789433521.958023:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433521.958028:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433521.958037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433521.958049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913735296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433521.958054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913735168:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433521.958058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433521.958062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433521.958063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433521.958066:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5891 00000100:00080000:1.0:1789433521.958068:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433521.958072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433521.958076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433521.958084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913735424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433521.958098:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433521.958100:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433521.958101:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433521.958103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433521.958105:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5891 00000100:00080000:1.0:1789433521.958108:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433521.958111:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433521.958115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433521.958124:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913735552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433521.958139:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433521/real 1789433521] req@ffff9811d0bdea00 x1876350913735168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433521.958156:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913735168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433521.958167:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433521.958171:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433521.958174:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913735168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433521.958193:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433521.958209:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433521.958217:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433521/real 1789433521] req@ffff9811d0bded80 x1876350913735296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433521.958225:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913735296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433521.958230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433521.958231:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433521.958233:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433521.958236:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433521.958245:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433521.958250:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433521/real 1789433521] req@ffff9811d0bdc000 x1876350913735424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433521.958256:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913735424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433521.958264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433521.958266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433521.958268:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433521.958271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433521.958281:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433521.958285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433521/real 1789433521] req@ffff9811d0bdd880 x1876350913735552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433521.958291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913735552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433521.958299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433521.958302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433521.958304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433522.854039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433523.878002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433524.902025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433525.926089:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433526.949987:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433527.077977:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433527.077987:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433527.077992:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433527.078001:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5896 00000100:00080000:0.0:1789433527.078010:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433527.078021:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433527.078055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913735680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433527.078091:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433527.078093:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433527.078095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433527.078100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5896 00000100:00080000:0.0:1789433527.078103:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433527.078107:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433527.078114:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433527.078123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913735808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433527.078131:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433527.078133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433527.078134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433527.078138:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5896 00000100:00100000:3.0:1789433527.078140:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913735680:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433527.078141:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433527.078145:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433527.078151:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433527.078157:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913735936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433527.078164:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433527.078166:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433527.078167:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433527.078170:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5896 00000100:00080000:0.0:1789433527.078172:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433527.078175:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433527.078178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433527.078184:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913736064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433527.078237:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433527.078262:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433527.078273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433527.974015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433528.997965:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433530.022013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789433530.150164:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:2.0:1789433530.150204:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789433530.150218:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433530.150263:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433530.150274:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433530.150281:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433530.150355:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433527/real 1789433530] req@ffff9811c8cd7b80 x1876350913735808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433530.150378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913735808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433530.150389:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433530.150393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433530.150397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433530.150406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433527/real 1789433530] req@ffff9811c8cd6300 x1876350913735936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433530.150417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913735936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433530.150423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433530.150427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433530.150429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913735936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433530.150436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433527/real 1789433530] req@ffff9811c8cd4000 x1876350913736064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433530.150447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913736064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433530.150455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433530.150458:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433530.150461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736064:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433530.150467:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433527/real 1789433530] req@ffff9811c8cd6a00 x1876350913735680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433530.150478:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913735680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433530.150485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433530.150488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433530.150490:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913735680:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789433531.046031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433532.070010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433532.197969:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433532.197981:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433532.197986:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433532.197997:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5901 00000100:00080000:0.0:1789433532.198006:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433532.198017:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433532.198055:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913736192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433532.198079:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433532.198084:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433532.198090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433532.198094:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5901 00000100:00080000:0.0:1789433532.198098:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433532.198103:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433532.198113:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433532.198122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913736192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433532.198132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913736320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433532.198147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433532.198150:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433532.198152:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433532.198157:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5901 00000100:00080000:0.0:1789433532.198161:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433532.198166:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433532.198171:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433532.198183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913736448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433532.198194:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433532.198197:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433532.198199:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433532.198203:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5901 00000100:00080000:0.0:1789433532.198208:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433532.198213:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433532.198219:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789433532.198221:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433532.198239:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913736576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433532.198288:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433532/real 1789433532] req@ffff9811c8cd5180 x1876350913736192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433532.198310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913736192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433532.198324:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433532.198329:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433532.198333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913736192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433532.198374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433532.198413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433532.198425:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433532/real 1789433532] req@ffff9811c8cd7800 x1876350913736320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433532.198437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913736320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433532.198446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433532.198450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433532.198453:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433532.198460:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433532.198488:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433532.198499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433532/real 1789433532] req@ffff9811c8cd7480 x1876350913736448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433532.198511:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913736448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433532.198519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433532.198522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433532.198525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433532.198532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433532.198552:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433532.198564:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433532/real 1789433532] req@ffff9811c8cd7100 x1876350913736576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433532.198576:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913736576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433532.198584:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433532.198587:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433532.198590:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433533.093996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433534.118018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433535.142016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433536.166021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433537.189973:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433537.317947:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433537.317954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433537.317958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433537.317968:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5906 00000100:00080000:3.0:1789433537.317976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433537.317988:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433537.318018:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913736704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433537.318042:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433537.318045:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433537.318046:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433537.318049:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5906 00000100:00080000:3.0:1789433537.318051:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433537.318054:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433537.318061:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433537.318069:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913736832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433537.318076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433537.318080:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789433537.318082:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913736704:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433537.318082:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433537.318096:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5906 00000100:00080000:3.0:1789433537.318101:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433537.318106:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433537.318111:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433537.318123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913736960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433537.318130:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433537.318134:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433537.318135:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433537.318139:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5906 00000800:00000100:0.0:1789433537.318144:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433537.318146:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433537.318153:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433537.318158:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433537.318170:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913737088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433537.318185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433537/real 1789433537] req@ffff9811d1bc4700 x1876350913736704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433537.318210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913736704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433537.318217:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433537.318220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433537.318222:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913736704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433537.318237:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433537.318258:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433537.318266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433537/real 1789433537] req@ffff9811d1bc6680 x1876350913736832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433537.318273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913736832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433537.318278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433537.318280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433537.318282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433537.318285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433537.318295:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433537.318300:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433537/real 1789433537] req@ffff9811d1bc5500 x1876350913736960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433537.318306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913736960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433537.318315:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433537.318316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433537.318318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913736960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433537.318321:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433537.318330:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433537.318334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433537/real 1789433537] req@ffff9811d1bc6300 x1876350913737088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433537.318340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913737088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433537.318345:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433537.318347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433537.318348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433538.213999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433539.237985:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433540.262005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433541.285993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433542.309995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433542.437944:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433542.437954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433542.437958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433542.437967:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5912 00000100:00080000:0.0:1789433542.437975:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433542.437986:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433542.438019:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913737216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433542.438050:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433542.438054:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433542.438055:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433542.438059:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5912 00000100:00080000:0.0:1789433542.438063:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433542.438069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433542.438078:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433542.438087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913737216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433542.438093:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913737344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433542.438105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433542.438108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433542.438110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433542.438113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5912 00000100:00080000:0.0:1789433542.438117:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433542.438121:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433542.438126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789433542.438134:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433542.438139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913737472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433542.438151:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433542.438154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433542.438156:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433542.438160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5912 00000100:00080000:0.0:1789433542.438165:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433542.438172:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433542.438174:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433542/real 1789433542] req@ffff9811c8cd6680 x1876350913737216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789433542.438178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433542.438195:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913737600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433542.438200:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913737216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433542.438220:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433542.438223:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433542.438226:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913737216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433542.438244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433542.438259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433542.438266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433542/real 1789433542] req@ffff9811c8cd4700 x1876350913737344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433542.438274:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913737344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433542.438279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433542.438281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433542.438283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433542.438286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433542.438298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433542.438307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433542/real 1789433542] req@ffff9811c8cd5c00 x1876350913737472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433542.438318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913737472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433542.438326:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433542.438329:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433542.438332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433542.438337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737600:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433542.438356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433542.438366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433542/real 1789433542] req@ffff9811c8cd7100 x1876350913737600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433542.438378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913737600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433542.438387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433542.438390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433542.438393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433543.333978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433544.357969:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433545.382000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433546.405980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433547.430005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433547.557929:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433547.557937:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433547.557941:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433547.557950:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5917 00000100:00080000:3.0:1789433547.557959:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433547.557981:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433547.558014:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913737728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433547.558036:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433547.558038:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433547.558040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433547.558042:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5917 00000100:00080000:3.0:1789433547.558045:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433547.558049:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433547.558056:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433547.558069:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913737856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433547.558079:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433547.558084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913737728:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433547.558084:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433547.558087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433547.558094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5917 00000100:00080000:3.0:1789433547.558098:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433547.558103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433547.558108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433547.558119:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913737984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433547.558130:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433547.558133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433547.558134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433547.558137:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5917 00000100:00080000:3.0:1789433547.558140:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433547.558143:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433547.558148:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789433547.558149:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433547.558158:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913738112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433547.558190:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433547/real 1789433547] req@ffff9811d1bc7100 x1876350913737728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433547.558209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913737728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433547.558219:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433547.558223:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433547.558226:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913737728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433547.558249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433547.558269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433547.558277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433547/real 1789433547] req@ffff9811d1bc6300 x1876350913737856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433547.558285:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913737856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433547.558290:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433547.558293:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433547.558295:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433547.558298:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433547.558308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433547.558322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433547/real 1789433547] req@ffff9811d1bc5500 x1876350913737984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433547.558334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913737984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433547.558344:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433547.558348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433547.558351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913737984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433547.558356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433547.558375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433547.558386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433547/real 1789433547] req@ffff9811d1bc6680 x1876350913738112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433547.558396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913738112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433547.558403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433547.558405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433547.558407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433548.453979:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433549.478001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433550.501982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433551.526037:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433552.549985:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433552.677945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433552.677953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433552.677956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433552.677965:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5922 00000100:00080000:1.0:1789433552.677973:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433552.677982:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433552.678013:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913738240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433552.678038:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433552.678041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433552.678043:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433552.678046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5922 00000100:00080000:1.0:1789433552.678049:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433552.678052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433552.678062:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433552.678072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913738368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433552.678079:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913738240:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433552.678085:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433552.678089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433552.678090:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433552.678094:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5922 00000100:00080000:1.0:1789433552.678097:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433552.678101:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433552.678105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433552.678115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913738496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433552.678129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433552.678132:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433552.678137:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433552.678139:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433552.678146:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5922 00000100:00080000:1.0:1789433552.678150:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433552.678155:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433552.678160:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433552.678165:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433552/real 1789433552] req@ffff9811d0bdf100 x1876350913738240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433552.678174:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913738624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433552.678184:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913738240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433552.678197:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433552.678201:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433552.678205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913738240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433552.678226:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433552.678248:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433552.678260:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433552/real 1789433552] req@ffff9811d0bde300 x1876350913738368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433552.678271:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913738368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433552.678278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433552.678280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433552.678283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433552.678289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433552.678308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433552.678318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433552/real 1789433552] req@ffff9811d0bdf800 x1876350913738496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433552.678331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913738496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433552.678339:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433552.678341:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433552.678344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433552.678349:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738624:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433552.678367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433552.678377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433552/real 1789433552] req@ffff9811d0bdd880 x1876350913738624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433552.678388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913738624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433552.678394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433552.678396:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433552.678398:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433553.574007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433554.598001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433555.621999:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433556.646007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433557.670018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433557.797947:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433557.797958:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433557.797963:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433557.797970:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5927 00000100:00080000:3.0:1789433557.797977:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433557.797985:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433557.798014:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913738752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433557.798034:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433557.798037:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433557.798038:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433557.798041:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5927 00000100:00080000:3.0:1789433557.798043:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433557.798045:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433557.798052:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433557.798063:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913738880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433557.798074:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433557.798075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913738752:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433557.798080:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433557.798085:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433557.798089:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5927 00000100:00080000:3.0:1789433557.798093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433557.798097:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433557.798103:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433557.798115:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913739008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433557.798126:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433557.798129:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433557.798131:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433557.798134:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5927 00000100:00080000:3.0:1789433557.798138:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433557.798142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433557.798146:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433557.798152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433557.798166:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913739136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433557.798184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433557/real 1789433557] req@ffff9811d1bc4700 x1876350913738752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433557.798203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913738752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433557.798215:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433557.798219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433557.798223:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913738752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433557.798245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433557.798267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433557.798280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433557/real 1789433557] req@ffff9811d1bc5880 x1876350913738880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433557.798291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913738880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433557.798299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433557.798302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433557.798305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913738880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433557.798311:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433557.798329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433557.798340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433557/real 1789433557] req@ffff9811d1bc5f80 x1876350913739008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433557.798352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913739008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433557.798360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433557.798363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433557.798366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433557.798373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433557.798390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433557.798400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433557/real 1789433557] req@ffff9811d1bc4000 x1876350913739136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433557.798411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913739136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433557.798419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433557.798422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433557.798426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433558.693935:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433559.717996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433560.741972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433561.765982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433562.790080:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433562.917957:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433562.917968:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433562.917973:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433562.917984:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5932 00000100:00080000:3.0:1789433562.917994:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433562.918009:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433562.918055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913739264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433562.918082:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433562.918085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433562.918087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433562.918090:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5932 00000100:00080000:3.0:1789433562.918093:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433562.918096:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433562.918105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433562.918114:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913739392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433562.918121:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433562.918123:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433562.918124:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433562.918127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5932 00000100:00080000:3.0:1789433562.918129:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433562.918131:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433562.918134:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433562.918139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913739520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433562.918154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433562.918157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433562.918158:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433562.918160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5932 00000100:00080000:3.0:1789433562.918162:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433562.918164:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433562.918173:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433562.918184:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913739264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433562.918186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913739648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433562.918278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433562.918324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433562/real 1789433562] req@ffff9811d1bc5c00 x1876350913739264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433562.918350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913739264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433562.918361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433562.918365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433562.918369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913739264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433562.918376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433562.918404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433562.918416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433562/real 1789433562] req@ffff9811d1bc7800 x1876350913739392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433562.918427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913739392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433562.918435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433562.918438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433562.918441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433562.918469:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433562.918489:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433562.918500:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433562/real 1789433562] req@ffff9811d1bc5180 x1876350913739520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433562.918511:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913739520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433562.918518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433562.918522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433562.918525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433562.918530:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433562.918547:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433562.918554:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433562/real 1789433562] req@ffff9811d1bc6a00 x1876350913739648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433562.918565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913739648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433562.918574:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433562.918577:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433562.918580:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433563.814089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433564.838023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433565.862019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433566.885986:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433567.909986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433568.037996:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433568.038009:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433568.038014:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433568.038025:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5937 00000100:00080000:0.0:1789433568.038037:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433568.038053:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433568.038100:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913739776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433568.038129:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433568.038133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433568.038135:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433568.038139:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5937 00000100:00080000:0.0:1789433568.038143:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433568.038148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433568.038159:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433568.038173:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913739904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433568.038178:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913739776:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433568.038187:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433568.038192:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433568.038194:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433568.038198:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5937 00000100:00080000:0.0:1789433568.038201:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433568.038205:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433568.038210:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433568.038221:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913740032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433568.038231:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433568.038234:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433568.038236:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433568.038239:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5937 00000100:00080000:0.0:1789433568.038247:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433568.038252:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433568.038257:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789433568.038266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433568.038270:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913740160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433568.038330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433568/real 1789433568] req@ffff9811c8cd7480 x1876350913739776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433568.038353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913739776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433568.038365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433568.038369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433568.038374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913739776:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433568.038401:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433568.038434:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433568.038449:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433568/real 1789433568] req@ffff9811c8cd7800 x1876350913739904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433568.038462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913739904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433568.038470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433568.038474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433568.038478:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913739904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433568.038484:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433568.038515:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433568.038527:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433568/real 1789433568] req@ffff9811c8cd5180 x1876350913740032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433568.038540:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913740032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433568.038548:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433568.038552:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433568.038555:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433568.038561:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740160:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433568.038592:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433568.038604:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433568/real 1789433568] req@ffff9811c8cd6a00 x1876350913740160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433568.038627:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913740160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433568.038643:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433568.038646:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433568.038649:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433568.934063:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433569.958061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433570.981996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433572.005974:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433573.030000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433573.157964:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433573.157972:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433573.157976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433573.157985:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5942 00000100:00080000:1.0:1789433573.157992:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433573.158002:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433573.158036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913740288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433573.158065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433573.158068:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433573.158070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433573.158074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5942 00000100:00080000:1.0:1789433573.158078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433573.158082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433573.158091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433573.158104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913740416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433573.158115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433573.158120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1789433573.158121:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913740288:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433573.158122:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433573.158136:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5942 00000100:00080000:1.0:1789433573.158144:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433573.158150:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433573.158155:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433573.158168:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913740544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433573.158185:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433573.158188:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433573.158190:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789433573.158191:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433573.158197:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5942 00000100:00080000:1.0:1789433573.158201:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433573.158207:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433573.158213:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433573.158225:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913740672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433573.158236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433573/real 1789433573] req@ffff9811d0bdc000 x1876350913740288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433573.158255:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913740288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433573.158266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433573.158269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433573.158272:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913740288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433573.158291:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433573.158310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433573.158320:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433573/real 1789433573] req@ffff9811d0bded80 x1876350913740416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433573.158331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913740416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433573.158338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433573.158341:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433573.158344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433573.158350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433573.158365:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433573.158374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433573/real 1789433573] req@ffff9811d0bdea00 x1876350913740544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433573.158384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913740544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433573.158390:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433573.158393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433573.158395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433573.158400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433573.158414:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433573.158423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433573/real 1789433573] req@ffff9811d0bdc700 x1876350913740672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433573.158433:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913740672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433573.158441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433573.158444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433573.158447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433574.054015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433575.078052:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433576.101998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433577.126003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433578.149991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433578.277944:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433578.277952:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433578.277956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433578.277965:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5947 00000100:00080000:1.0:1789433578.277972:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433578.277981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433578.278007:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913740800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433578.278026:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433578.278028:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433578.278032:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433578.278035:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5947 00000100:00080000:1.0:1789433578.278037:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433578.278040:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433578.278047:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433578.278056:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913740928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433578.278063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433578.278065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433578.278066:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433578.278069:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5947 00000100:00080000:1.0:1789433578.278071:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433578.278073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913740800:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433578.278076:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433578.278083:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433578.278089:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913741056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433578.278095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433578.278098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433578.278099:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433578.278101:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5947 00000100:00080000:1.0:1789433578.278103:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433578.278105:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433578.278108:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433578.278114:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913741184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433578.278139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433578.278169:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433578/real 1789433578] req@ffff9811d0bdfb80 x1876350913740800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433578.278183:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913740800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433578.278190:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433578.278192:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433578.278195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913740800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433578.278210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433578.278225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433578.278232:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433578/real 1789433578] req@ffff9811d0bdf480 x1876350913740928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433578.278239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913740928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433578.278244:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433578.278246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433578.278247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913740928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433578.278250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433578.278259:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433578.278264:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433578/real 1789433578] req@ffff9811d0bde680 x1876350913741056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433578.278270:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913741056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433578.278278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433578.278280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433578.278282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433578.278285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741184:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433578.278294:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433578.278299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433578/real 1789433578] req@ffff9811d0bdd500 x1876350913741184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433578.278305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913741184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433578.278309:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433578.278311:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433578.278313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433579.174067:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433580.198054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433581.221981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433582.245996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433583.269970:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433583.397940:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433583.397948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433583.397952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433583.397960:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5952 00000100:00080000:0.0:1789433583.397966:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433583.397977:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433583.398010:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913741312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433583.398037:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433583.398040:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433583.398042:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433583.398046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5952 00000100:00080000:0.0:1789433583.398050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433583.398054:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433583.398063:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433583.398077:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913741440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433583.398080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913741312:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433583.398095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433583.398098:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433583.398100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433583.398111:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5952 00000100:00080000:0.0:1789433583.398118:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433583.398124:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433583.398129:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789433583.398138:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433583.398143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913741568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433583.398160:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433583.398163:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433583.398165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433583.398169:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5952 00000100:00080000:0.0:1789433583.398173:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433583.398176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433583/real 1789433583] req@ffff9811c8cd4000 x1876350913741312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433583.398180:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433583.398196:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433583.398198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913741312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433583.398211:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789433583.398212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913741696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433583.398218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433583.398225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913741312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433583.398246:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433583.398262:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433583.398270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433583/real 1789433583] req@ffff9811c8cd6300 x1876350913741440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433583.398277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913741440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433583.398282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433583.398284:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433583.398286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433583.398289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433583.398298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433583.398302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433583/real 1789433583] req@ffff9811c8cd7b80 x1876350913741568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433583.398309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913741568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433583.398313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433583.398314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433583.398316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433583.398319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433583.398328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433583.398334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433583/real 1789433583] req@ffff9811c8cd4a80 x1876350913741696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433583.398345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913741696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433583.398352:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433583.398355:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433583.398358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433584.293988:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433585.318006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433586.342220:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433587.366006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433588.389984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433588.517936:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433588.517944:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433588.517948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433588.517955:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5958 00000100:00080000:3.0:1789433588.517961:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433588.517969:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433588.517998:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913741824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433588.518019:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433588.518022:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433588.518023:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433588.518026:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5958 00000100:00080000:3.0:1789433588.518028:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433588.518030:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433588.518037:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433588.518046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913741952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433588.518058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433588.518062:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433588.518063:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433588.518064:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913741824:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433588.518066:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5958 00000100:00080000:3.0:1789433588.518072:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433588.518082:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433588.518087:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433588.518099:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913742080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433588.518110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433588.518114:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433588.518116:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433588.518119:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5958 00000100:00080000:3.0:1789433588.518123:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433588.518128:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433588.518131:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433588.518137:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433588.518145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913742208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433588.518170:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433588/real 1789433588] req@ffff9811d1bc7480 x1876350913741824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433588.518188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913741824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433588.518198:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433588.518202:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433588.518205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913741824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433588.518234:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433588.518257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433588.518270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433588/real 1789433588] req@ffff9811d1bc6a00 x1876350913741952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433588.518291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913741952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433588.518300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433588.518303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433588.518306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913741952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433588.518312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433588.518340:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433588.518350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433588/real 1789433588] req@ffff9811d1bc5180 x1876350913742080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433588.518360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913742080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433588.518367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433588.518370:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433588.518372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433588.518378:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433588.518395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433588.518405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433588/real 1789433588] req@ffff9811d1bc7800 x1876350913742208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433588.518416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913742208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433588.518424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433588.518427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433588.518431:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433589.414077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433590.438004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433591.462052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433592.486121:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433593.510003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433593.637943:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433593.637954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433593.637957:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433593.637967:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5963 00000100:00080000:3.0:1789433593.637976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433593.637986:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433593.638031:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913742336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433593.638060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433593.638063:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433593.638065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433593.638069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5963 00000100:00080000:3.0:1789433593.638073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433593.638077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433593.638085:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433593.638097:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913742464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433593.638105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433593.638108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433593.638110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433593.638113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5963 00000100:00080000:3.0:1789433593.638116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433593.638119:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433593.638124:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433593.638127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913742336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433593.638132:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913742592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433593.638141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433593.638144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433593.638145:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433593.638148:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5963 00000100:00080000:3.0:1789433593.638151:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433593.638154:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433593.638158:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433593.638166:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913742720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433593.638208:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433593.638238:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433593.638252:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433594.534088:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433595.558035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433596.582006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433596.710130:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433596.710178:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433596.710193:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433596.710224:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433596.710232:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433596.710252:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433596.710276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433593/real 1789433596] req@ffff9811d1bc5f80 x1876350913742592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433596.710306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913742592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433596.710318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433596.710323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433596.710328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433596.710335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433593/real 1789433596] req@ffff9811d1bc5880 x1876350913742720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433596.710347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913742720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433596.710355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433596.710358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433596.710361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742720:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433596.710368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433593/real 1789433596] req@ffff9811d1bc5c00 x1876350913742336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433596.710380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913742336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433596.710386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433596.710390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433596.710393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913742336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433596.710399:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433593/real 1789433596] req@ffff9811d1bc4000 x1876350913742464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433596.710441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913742464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433596.710449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433596.710453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433596.710455:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742464:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789433597.605993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433598.629979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433598.757952:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433598.757963:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433598.757968:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433598.757984:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5968 00000100:00080000:3.0:1789433598.757996:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433598.758013:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433598.758057:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913742848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433598.758102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433598.758107:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433598.758109:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433598.758113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5968 00000100:00080000:3.0:1789433598.758117:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433598.758123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433598.758135:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433598.758154:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913742848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433598.758162:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913742976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433598.758175:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433598.758179:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433598.758181:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433598.758185:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5968 00000100:00080000:3.0:1789433598.758190:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433598.758195:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433598.758200:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433598.758213:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913743104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433598.758223:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433598.758227:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433598.758229:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433598.758232:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5968 00000100:00080000:3.0:1789433598.758237:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433598.758242:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433598.758245:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433598.758256:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433598.758268:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913743232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433598.758309:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433598/real 1789433598] req@ffff9811d1bc4000 x1876350913742848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433598.758329:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913742848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433598.758341:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433598.758346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433598.758349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913742848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433598.758374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433598.758410:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433598.758424:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433598/real 1789433598] req@ffff9811d1bc5c00 x1876350913742976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433598.758434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913742976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433598.758441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433598.758444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433598.758447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913742976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433598.758452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433598.758470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433598.758479:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433598/real 1789433598] req@ffff9811d1bc5880 x1876350913743104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433598.758488:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913743104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433598.758495:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433598.758498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433598.758501:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433598.758508:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433598.758528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433598.758538:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433598/real 1789433598] req@ffff9811d1bc5f80 x1876350913743232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433598.758549:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913743232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433598.758558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433598.758561:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433598.758564:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433599.654002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433600.678007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433601.701998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433602.725967:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433603.750083:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433603.877945:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433603.877955:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433603.877959:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433603.877970:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5973 00000100:00080000:0.0:1789433603.877980:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433603.877992:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433603.878021:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913743360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433603.878043:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433603.878046:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433603.878047:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433603.878050:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5973 00000100:00080000:0.0:1789433603.878053:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433603.878057:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433603.878071:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433603.878073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913743360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433603.878084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913743488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433603.878092:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433603.878094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433603.878096:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433603.878098:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5973 00000100:00080000:0.0:1789433603.878100:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433603.878103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433603.878105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433603.878112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913743616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433603.878120:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433603.878122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433603.878123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789433603.878128:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433603.878131:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5973 00000100:00080000:0.0:1789433603.878134:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433603.878138:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433603.878141:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433603.878150:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913743744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433603.878162:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433603/real 1789433603] req@ffff9811c8cd5500 x1876350913743360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433603.878176:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913743360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433603.878186:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433603.878190:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433603.878193:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913743360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433603.878212:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433603.878228:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433603.878236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433603/real 1789433603] req@ffff9811c8cd6d80 x1876350913743488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433603.878243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913743488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433603.878248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433603.878250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433603.878253:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433603.878256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433603.878265:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433603.878270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433603/real 1789433603] req@ffff9811c8cd5f80 x1876350913743616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433603.878277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913743616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433603.878281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433603.878282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433603.878284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433603.878287:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433603.878296:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433603.878302:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433603/real 1789433603] req@ffff9811c8cd4a80 x1876350913743744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433603.878312:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913743744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433603.878320:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433603.878323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433603.878326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913743744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433604.774054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433605.798037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433606.822030:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433607.845972:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433608.870054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433608.997968:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433608.997978:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433608.997982:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433608.997991:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5978 00000100:00080000:3.0:1789433608.997999:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433608.998010:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433608.998044:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913743872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433608.998084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433608.998088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433608.998090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433608.998094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5978 00000100:00080000:3.0:1789433608.998097:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433608.998103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433608.998112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433608.998125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913744000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433608.998130:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913743872:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433608.998141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433608.998145:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433608.998158:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433608.998162:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5978 00000100:00080000:3.0:1789433608.998167:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433608.998171:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433608.998176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433608.998190:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913744128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433608.998194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433608.998205:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433608.998209:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433608.998211:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433608.998216:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5978 00000100:00080000:3.0:1789433608.998219:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433608.998224:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433608.998230:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433608.998237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433608/real 1789433608] req@ffff9811d1bc4700 x1876350913743872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433608.998244:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913744256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433608.998272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913743872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433608.998291:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433608.998306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433608.998310:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913743872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433608.998339:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433608.998363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433608.998377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433608/real 1789433608] req@ffff9811d1bc6680 x1876350913744000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433608.998390:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913744000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433608.998399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433608.998402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433608.998405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433608.998411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433608.998432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433608.998443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433608/real 1789433608] req@ffff9811d1bc5500 x1876350913744128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433608.998455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913744128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433608.998474:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433608.998478:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433608.998481:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433608.998487:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744256:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433608.998508:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433608.998520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433608/real 1789433608] req@ffff9811d1bc6300 x1876350913744256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433608.998530:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913744256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433608.998539:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433608.998542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433608.998545:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433609.894019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433610.918114:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433611.942023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433612.966005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433613.990047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433614.118000:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433614.118009:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433614.118013:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433614.118022:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5983 00000100:00080000:3.0:1789433614.118032:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433614.118041:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433614.118080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913744384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433614.118115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433614.118119:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433614.118121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433614.118125:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5983 00000100:00080000:3.0:1789433614.118129:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433614.118134:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433614.118142:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433614.118156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913744512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433614.118165:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433614.118168:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433614.118170:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433614.118173:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5983 00000100:00080000:3.0:1789433614.118177:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433614.118180:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433614.118184:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433614.118194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913744640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433614.118203:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433614.118206:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433614.118208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433614.118211:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5983 00000100:00080000:3.0:1789433614.118214:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433614.118218:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433614.118222:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433614.118230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913744768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433614.118441:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913744384:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433614.118521:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433614.118581:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433614/real 1789433614] req@ffff9811d1bc7100 x1876350913744384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433614.118609:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913744384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433614.118620:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433614.118625:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433614.118629:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913744384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433614.118637:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433614.118660:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433614.118673:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433614/real 1789433614] req@ffff9811d1bc6300 x1876350913744512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433614.118686:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913744512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433614.118695:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433614.118698:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433614.118702:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433614.118708:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433614.118726:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433614.118736:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433614/real 1789433614] req@ffff9811d1bc5500 x1876350913744640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433614.118748:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913744640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433614.118762:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433614.118766:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433614.118769:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433614.118775:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433614.118791:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433614.118800:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433614/real 1789433614] req@ffff9811d1bc6680 x1876350913744768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433614.118811:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913744768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433614.118820:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433614.118823:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433614.118827:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913744768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433615.014002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433616.037984:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433617.062023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433618.085986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433619.110001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433619.237951:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433619.237960:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433619.237964:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433619.237973:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5988 00000100:00080000:2.0:1789433619.237981:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433619.237992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433619.238025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913744896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433619.238049:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433619.238052:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433619.238053:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433619.238057:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5988 00000100:00080000:2.0:1789433619.238060:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433619.238064:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433619.238073:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433619.238085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913745024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433619.238094:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913744896:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433619.238097:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433619.238103:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433619.238105:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433619.238109:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5988 00000100:00080000:2.0:1789433619.238112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433619.238117:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433619.238121:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433619.238132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913745152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433619.238142:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433619.238145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433619.238147:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433619.238151:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5988 00000100:00080000:2.0:1789433619.238154:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433619.238159:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433619.238162:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789433619.238170:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433619.238182:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913745280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433619.238206:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433619/real 1789433619] req@ffff9811d2101500 x1876350913744896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433619.238224:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913744896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433619.238234:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433619.238252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433619.238256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913744896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433619.238278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433619.238301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433619.238314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433619/real 1789433619] req@ffff9811d2101c00 x1876350913745024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433619.238326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913745024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433619.238334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433619.238337:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433619.238340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433619.238346:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433619.238363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433619.238372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433619/real 1789433619] req@ffff9811d2103b80 x1876350913745152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433619.238383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913745152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433619.238389:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433619.238393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433619.238395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433619.238400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745280:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433619.238418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433619.238428:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433619/real 1789433619] req@ffff9811d2102a00 x1876350913745280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433619.238438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913745280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433619.238445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433619.238448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433619.238451:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433620.134019:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433621.158019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433622.182038:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433623.205988:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433624.230597:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433624.357982:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433624.357992:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433624.357996:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433624.358005:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5993 00000100:00080000:2.0:1789433624.358013:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433624.358024:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433624.358067:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913745408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433624.358100:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433624.358103:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433624.358105:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433624.358110:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5993 00000100:00080000:2.0:1789433624.358113:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433624.358118:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433624.358127:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433624.358141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913745536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433624.358151:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433624.358153:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433624.358155:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433624.358158:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5993 00000100:00080000:2.0:1789433624.358161:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433624.358165:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433624.358170:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433624.358181:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913745664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433624.358192:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913745408:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433624.358194:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433624.358213:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433624.358215:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433624.358219:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5993 00000100:00080000:2.0:1789433624.358223:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433624.358228:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433624.358234:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433624.358247:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913745792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433624.358275:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433624.358318:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433624/real 1789433624] req@ffff9811d2100000 x1876350913745408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433624.358337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913745408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433624.358349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433624.358352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433624.358356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913745408:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433624.358362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433624.358384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433624.358396:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433624/real 1789433624] req@ffff9811d2103800 x1876350913745536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433624.358413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913745536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433624.358421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433624.358424:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433624.358427:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433624.358449:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433624.358470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433624.358481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433624/real 1789433624] req@ffff9811d2102d80 x1876350913745664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433624.358492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913745664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433624.358499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433624.358502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433624.358505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433624.358510:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745792:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433624.358528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433624.358538:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433624/real 1789433624] req@ffff9811d2102a00 x1876350913745792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433624.358547:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913745792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433624.358560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433624.358563:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433624.358566:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913745792:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433625.253997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433626.277960:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433627.302007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433628.325989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433629.349967:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433629.477958:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433629.477967:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433629.477972:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433629.477981:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 5999 00000100:00080000:2.0:1789433629.477989:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433629.478000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433629.478042:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913745920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433629.478068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433629.478072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433629.478074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433629.478079:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5999 00000100:00080000:2.0:1789433629.478082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433629.478088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433629.478097:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433629.478110:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913746048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433629.478121:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433629.478123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913745920:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433629.478127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433629.478132:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433629.478137:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5999 00000100:00080000:2.0:1789433629.478141:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433629.478146:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433629.478151:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433629.478175:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913746176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433629.478192:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433629.478196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433629.478199:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433629.478200:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433629.478207:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 5999 00000100:00080000:2.0:1789433629.478213:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433629.478219:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433629.478224:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433629.478237:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913746304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433629.478271:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433629/real 1789433629] req@ffff9811d2103b80 x1876350913745920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433629.478289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913745920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433629.478299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433629.478304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433629.478307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913745920:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433629.478329:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433629.478352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433629.478364:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433629/real 1789433629] req@ffff9811d2101c00 x1876350913746048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433629.478376:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913746048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433629.478384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433629.478387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433629.478390:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433629.478396:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433629.478413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433629.478424:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433629/real 1789433629] req@ffff9811d2101500 x1876350913746176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433629.478435:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913746176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433629.478442:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433629.478445:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433629.478448:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433629.478467:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746304:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433629.478486:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433629.478497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433629/real 1789433629] req@ffff9811d2100e00 x1876350913746304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433629.478508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913746304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433629.478515:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433629.478518:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433629.478521:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746304:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433630.374029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433631.397995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433632.422054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433633.446023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433634.470121:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433634.597966:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433634.597976:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433634.597979:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433634.597988:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6004 00000100:00080000:3.0:1789433634.597996:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433634.598006:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433634.598042:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913746432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433634.598068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433634.598072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433634.598074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433634.598078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6004 00000100:00080000:3.0:1789433634.598082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433634.598087:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433634.598106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433634.598119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913746432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433634.598122:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913746560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433634.598147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433634.598150:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433634.598152:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433634.598156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6004 00000100:00080000:3.0:1789433634.598159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433634.598164:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433634.598169:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433634.598180:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913746688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433634.598191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433634.598194:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433634.598198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433634.598199:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433634.598204:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6004 00000100:00080000:3.0:1789433634.598209:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433634.598214:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433634.598219:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433634.598232:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913746816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433634.598246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433634/real 1789433634] req@ffff9811d1bc4700 x1876350913746432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433634.598266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913746432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433634.598276:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433634.598280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433634.598284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913746432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433634.598303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433634.598323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433634.598336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433634/real 1789433634] req@ffff9811d1bc5f80 x1876350913746560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433634.598347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913746560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433634.598355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433634.598358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433634.598361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433634.598366:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433634.598385:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433634.598396:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433634/real 1789433634] req@ffff9811d1bc5880 x1876350913746688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433634.598407:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913746688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433634.598414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433634.598417:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433634.598420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433634.598425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746816:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433634.598444:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433634.598453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433634/real 1789433634] req@ffff9811d1bc5c00 x1876350913746816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433634.598464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913746816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433634.598471:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433634.598474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433634.598477:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913746816:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433635.493957:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433636.518393:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433637.542026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433638.566029:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433639.589995:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433639.717889:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433639.717899:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433639.717903:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433639.717910:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6009 00000100:00080000:3.0:1789433639.717917:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433639.717932:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433639.717963:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913746944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433639.717989:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433639.717995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433639.717996:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433639.718000:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6009 00000100:00080000:3.0:1789433639.718003:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433639.718007:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433639.718016:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433639.718023:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913746944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433639.718028:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913747072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433639.718036:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433639.718038:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433639.718039:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433639.718041:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6009 00000100:00080000:3.0:1789433639.718043:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433639.718045:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433639.718048:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433639.718054:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913747200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433639.718061:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433639.718063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433639.718064:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433639.718067:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6009 00000100:00080000:3.0:1789433639.718070:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433639.718072:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433639.718075:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433639.718079:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433639.718085:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913747328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433639.718127:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433639/real 1789433639] req@ffff9811d1bc4000 x1876350913746944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433639.718141:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913746944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433639.718148:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433639.718150:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433639.718153:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913746944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433639.718169:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433639.718184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433639.718191:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433639/real 1789433639] req@ffff9811d1bc7800 x1876350913747072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433639.718198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913747072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433639.718204:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433639.718205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433639.718207:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433639.718210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433639.718220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433639.718225:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433639/real 1789433639] req@ffff9811d1bc5180 x1876350913747200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433639.718231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913747200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433639.718235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433639.718237:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433639.718238:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433639.718242:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747328:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433639.718251:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433639.718256:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433639/real 1789433639] req@ffff9811d1bc6a00 x1876350913747328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433639.718262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913747328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433639.718267:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433639.718268:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433639.718270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747328:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433640.613992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433641.637997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433642.662000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433643.686010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433644.710171:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433644.837931:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433644.837940:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433644.837944:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433644.837954:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6014 00000100:00080000:3.0:1789433644.837962:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433644.837971:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433644.838004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913747456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433644.838031:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433644.838035:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433644.838036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433644.838040:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6014 00000100:00080000:3.0:1789433644.838044:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433644.838060:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433644.838080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433644.838091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913747456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433644.838096:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913747584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433644.838108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433644.838112:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433644.838113:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433644.838117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6014 00000100:00080000:3.0:1789433644.838121:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433644.838125:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433644.838130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433644.838141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913747712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433644.838150:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433644.838155:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433644.838156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433644.838159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433644.838164:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6014 00000100:00080000:3.0:1789433644.838169:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433644.838174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433644.838179:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433644.838190:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913747840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433644.838218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433644/real 1789433644] req@ffff9811d1bc7480 x1876350913747456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433644.838238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913747456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433644.838249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433644.838263:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433644.838267:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913747456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433644.838290:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433644.838326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433644.838340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433644/real 1789433644] req@ffff9811d1bc6a00 x1876350913747584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433644.838352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913747584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433644.838361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433644.838364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433644.838368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433644.838374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433644.838393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433644.838405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433644/real 1789433644] req@ffff9811d1bc5180 x1876350913747712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433644.838415:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913747712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433644.838422:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433644.838425:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433644.838429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433644.838435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747840:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433644.838453:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433644.838464:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433644/real 1789433644] req@ffff9811d1bc7800 x1876350913747840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433644.838482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913747840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433644.838490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433644.838493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433644.838497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913747840:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433645.734033:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433646.757999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433647.781960:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433648.805985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433649.830060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433649.957924:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433649.957933:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433649.957937:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433649.957946:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6019 00000100:00080000:3.0:1789433649.957954:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433649.957963:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433649.957995:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913747968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433649.958030:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433649.958034:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433649.958036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433649.958040:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6019 00000100:00080000:3.0:1789433649.958049:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433649.958054:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433649.958063:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433649.958071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913747968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433649.958077:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913748096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433649.958088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433649.958091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433649.958093:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433649.958097:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6019 00000100:00080000:3.0:1789433649.958100:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433649.958103:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433649.958107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433649.958116:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913748224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433649.958131:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433649.958134:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433649.958138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433649.958140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433649.958144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6019 00000100:00080000:3.0:1789433649.958148:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433649.958151:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433649.958155:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433649.958167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913748352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433649.958190:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433649/real 1789433649] req@ffff9811d1bc4000 x1876350913747968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433649.958215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913747968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433649.958222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433649.958225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433649.958227:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913747968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433649.958244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433649.958260:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433649.958268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433649/real 1789433649] req@ffff9811d1bc5c00 x1876350913748096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433649.958283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913748096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433649.958289:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433649.958290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433649.958292:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433649.958296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433649.958306:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433649.958317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433649/real 1789433649] req@ffff9811d1bc5880 x1876350913748224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433649.958327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913748224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433649.958334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433649.958337:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433649.958340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433649.958346:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748352:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433649.958365:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433649.958374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433649/real 1789433649] req@ffff9811d1bc5f80 x1876350913748352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433649.958381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913748352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433649.958386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433649.958388:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433649.958390:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748352:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433650.854047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433651.878028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433652.902124:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433653.926022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433654.950092:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433655.078478:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433655.078487:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433655.078491:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433655.078501:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6024 00000100:00080000:3.0:1789433655.078509:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433655.078518:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433655.078551:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913748480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433655.078578:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433655.078581:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433655.078583:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433655.078622:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6024 00000100:00100000:0.0:1789433655.078628:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913748480:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433655.078629:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433655.078634:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433655.078644:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433655.078658:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913748608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433655.078669:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433655.078672:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433655.078674:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433655.078677:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6024 00000100:00080000:3.0:1789433655.078680:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433655.078685:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433655.078689:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433655.078699:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913748736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433655.078708:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433655.078709:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433655.078713:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433655.078716:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433655.078720:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6024 00000100:00080000:3.0:1789433655.078724:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433655.078728:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433655.078733:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433655.078744:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913748864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433655.078753:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433655/real 1789433655] req@ffff9811d1bc4700 x1876350913748480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433655.078779:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913748480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433655.078791:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433655.078796:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433655.078799:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913748480:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433655.078820:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433655.078878:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433655.078891:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433655/real 1789433655] req@ffff9811d1bc6680 x1876350913748608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433655.078903:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913748608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433655.078911:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433655.078915:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433655.078918:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433655.078923:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433655.078959:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433655.078971:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433655/real 1789433655] req@ffff9811d1bc5500 x1876350913748736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433655.078998:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913748736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433655.079005:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433655.079009:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433655.079012:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433655.079017:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748864:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433655.079037:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433655.079048:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433655/real 1789433655] req@ffff9811d1bc6300 x1876350913748864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433655.079060:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913748864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433655.079068:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433655.079072:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433655.079075:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913748864:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433655.975146:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433656.998014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433658.022482:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433659.046361:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433660.070339:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433660.198071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433660.198081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433660.198085:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433660.198094:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6029 00000100:00080000:2.0:1789433660.198102:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433660.198113:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433660.198147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913748992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433660.198177:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433660.198180:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433660.198182:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433660.198186:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6029 00000100:00080000:2.0:1789433660.198189:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433660.198194:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433660.198202:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433660.198213:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913749120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433660.198222:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433660.198225:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433660.198227:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433660.198231:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6029 00000100:00080000:2.0:1789433660.198234:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433660.198239:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433660.198244:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433660.198249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913748992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433660.198257:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913749248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433660.198269:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433660.198272:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433660.198274:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433660.198277:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6029 00000100:00080000:2.0:1789433660.198281:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433660.198286:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433660.198291:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433660.198302:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913749376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433660.198380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433660.198414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433660.198440:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749376:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433661.094464:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433662.118050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433663.141996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433663.270442:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433663.270492:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433663.270506:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433663.270556:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433663.270566:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433663.270573:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433663.270787:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433660/real 1789433663] req@ffff9811d2101f80 x1876350913749248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433663.270823:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913749248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433663.270874:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433663.270880:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433663.270884:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433663.270892:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433660/real 1789433663] req@ffff9811d2100e00 x1876350913749376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433663.270906:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913749376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433663.270925:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433663.270929:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433663.270932:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749376:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433663.270939:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433660/real 1789433663] req@ffff9811d2101880 x1876350913748992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433663.270955:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913748992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433663.270962:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433663.270966:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433663.270969:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913748992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433663.270975:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433660/real 1789433663] req@ffff9811d2101180 x1876350913749120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433663.270985:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913749120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433663.270997:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433663.271000:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433663.271003:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749120:192.168.203.128@tcp:38:kworker.0 00010000:00010000:3.0:1789433664.166025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433665.190016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433665.317945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433665.317953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433665.317957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433665.317966:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6034 00000100:00080000:3.0:1789433665.318023:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433665.318033:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433665.318068:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913749504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433665.318107:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433665.318111:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433665.318112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433665.318117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6034 00000100:00080000:3.0:1789433665.318120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433665.318126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433665.318136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433665.318149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913749632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433665.318166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433665.318172:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433665.318174:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433665.318177:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6034 00000100:00080000:3.0:1789433665.318180:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433665.318184:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433665.318189:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433665.318199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913749760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433665.318208:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433665.318224:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433665.318226:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433665.318229:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6034 00000100:00080000:3.0:1789433665.318240:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433665.318245:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433665.318250:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433665.318266:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913749888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433665.319996:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913749504:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433665.320104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433665.320156:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433665/real 1789433665] req@ffff9811d1bc7100 x1876350913749504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433665.320176:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913749504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433665.320186:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433665.320190:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433665.320194:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913749504:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433665.320201:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433665.320223:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433665.320235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433665/real 1789433665] req@ffff9811d1bc6300 x1876350913749632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433665.320247:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913749632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433665.320254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433665.320258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433665.320260:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433665.320266:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433665.320285:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433665.320295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433665/real 1789433665] req@ffff9811d1bc5500 x1876350913749760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433665.320306:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913749760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433665.320313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433665.320316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433665.320319:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433665.320325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749888:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433665.320344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433665.320354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433665/real 1789433665] req@ffff9811d1bc6680 x1876350913749888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433665.320366:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913749888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433665.320379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433665.320382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433665.320385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913749888:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433666.214022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433667.238025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433668.261993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433669.286011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433670.310016:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433670.437968:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433670.437977:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433670.437981:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433670.437991:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6040 00000100:00080000:2.0:1789433670.437999:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433670.438010:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433670.438042:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913750016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433670.438068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433670.438072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433670.438074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433670.438078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6040 00000100:00080000:2.0:1789433670.438081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433670.438086:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433670.438095:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433670.438113:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913750144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433670.438123:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433670.438126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433670.438128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433670.438131:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6040 00000100:00080000:2.0:1789433670.438134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433670.438139:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433670.438144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433670.438146:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913750016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433670.438167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913750272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433670.438179:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433670.438182:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433670.438184:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433670.438188:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6040 00000100:00080000:2.0:1789433670.438192:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433670.438199:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433670.438204:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433670.438208:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433670.438220:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913750400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433670.438246:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433670/real 1789433670] req@ffff9811d2101500 x1876350913750016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433670.438263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913750016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433670.438273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433670.438278:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433670.438281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913750016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433670.438288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433670.438308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433670.438317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433670/real 1789433670] req@ffff9811d2101c00 x1876350913750144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433670.438324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913750144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433670.438329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433670.438331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433670.438334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433670.438350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433670.438360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433670.438366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433670/real 1789433670] req@ffff9811d2103b80 x1876350913750272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433670.438372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913750272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433670.438377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433670.438378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433670.438380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433670.438383:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750400:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433670.438392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433670.438397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433670/real 1789433670] req@ffff9811d2102a00 x1876350913750400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433670.438403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913750400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433670.438408:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433670.438409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433670.438411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750400:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433671.334031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433672.358078:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433673.382046:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433674.406070:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433675.430016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433675.558081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433675.558090:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433675.558095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433675.558106:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6045 00000100:00080000:1.0:1789433675.558114:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433675.558124:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433675.558156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913750528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433675.558182:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433675.558185:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433675.558187:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433675.558191:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6045 00000100:00080000:1.0:1789433675.558194:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433675.558198:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433675.558207:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433675.558218:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913750656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433675.558228:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433675.558233:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789433675.558234:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913750528:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433675.558235:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433675.558243:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6045 00000100:00080000:1.0:1789433675.558246:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433675.558250:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433675.558255:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433675.558265:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913750784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433675.558287:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433675.558290:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433675.558292:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433675.558301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433675.558349:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6045 00000100:00100000:0.0:1789433675.558353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433675/real 1789433675] req@ffff9811d0bdd180 x1876350913750528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433675.558355:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433675.558363:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433675.558371:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433675.558380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913750528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433675.558386:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913750912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433675.558398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433675.558405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433675.558410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913750528:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433675.558432:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433675.558455:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433675.558467:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433675/real 1789433675] req@ffff9811d0bdc380 x1876350913750656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433675.558477:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913750656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433675.558485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433675.558488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433675.558490:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433675.558496:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433675.558513:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433675.558523:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433675/real 1789433675] req@ffff9811d0bddc00 x1876350913750784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433675.558540:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913750784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433675.558548:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433675.558551:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433675.558554:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433675.558559:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750912:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433675.558574:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433675.558582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433675/real 1789433675] req@ffff9811d0bdd500 x1876350913750912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433675.558592:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913750912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433675.558600:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433675.558603:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433675.558606:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913750912:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433676.454004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433677.478160:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433678.502018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433679.525992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433680.550058:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433680.677980:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433680.677989:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433680.677994:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433680.678004:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6050 00000100:00080000:2.0:1789433680.678012:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433680.678022:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433680.678061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913751040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433680.678087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433680.678089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433680.678091:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433680.678094:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6050 00000100:00080000:2.0:1789433680.678096:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433680.678098:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433680.678107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433680.678115:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913751168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433680.678122:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433680.678125:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433680.678126:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433680.678128:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6050 00000100:00080000:2.0:1789433680.678130:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433680.678132:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433680.678136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433680.678142:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913751040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433680.678144:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913751296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433680.678155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433680.678157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433680.678158:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433680.678160:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6050 00000100:00080000:2.0:1789433680.678162:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433680.678165:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433680.678168:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433680.678178:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913751424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433680.678210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433680.678247:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433680/real 1789433680] req@ffff9811d2102d80 x1876350913751040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433680.678260:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913751040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433680.678270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433680.678273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433680.678276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913751040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433680.678281:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433680.678294:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433680.678301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433680/real 1789433680] req@ffff9811d2103800 x1876350913751168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433680.678308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913751168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433680.678313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433680.678315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433680.678317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433680.678337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433680.678346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433680.678351:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433680/real 1789433680] req@ffff9811d2100000 x1876350913751296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433680.678357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913751296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433680.678362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433680.678364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433680.678366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433680.678370:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751424:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433680.678379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433680.678384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433680/real 1789433680] req@ffff9811d2102a00 x1876350913751424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433680.678391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913751424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433680.678396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433680.678397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433680.678399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751424:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433681.574061:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433682.597975:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433683.621982:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433684.645953:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433685.669985:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433685.797913:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433685.797921:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433685.797924:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433685.797932:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6055 00000100:00080000:2.0:1789433685.797939:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433685.797948:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433685.797976:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913751552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433685.797996:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433685.797998:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433685.798000:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433685.798002:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6055 00000100:00080000:2.0:1789433685.798005:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433685.798007:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433685.798015:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433685.798022:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913751680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433685.798029:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913751552:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433685.798032:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433685.798036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433685.798037:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433685.798040:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6055 00000100:00080000:2.0:1789433685.798042:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433685.798044:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433685.798047:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433685.798053:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913751808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433685.798060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433685.798062:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433685.798064:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433685.798066:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6055 00000100:00080000:2.0:1789433685.798068:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433685.798077:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433685.798081:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433685.798085:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433685.798092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913751936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433685.798131:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433685/real 1789433685] req@ffff9811d2103b80 x1876350913751552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433685.798144:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913751552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433685.798152:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433685.798156:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433685.798159:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913751552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433685.798180:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433685.798196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433685.798204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433685/real 1789433685] req@ffff9811d2101c00 x1876350913751680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433685.798211:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913751680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433685.798216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433685.798218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433685.798220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433685.798223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433685.798232:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433685.798237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433685/real 1789433685] req@ffff9811d2101500 x1876350913751808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433685.798243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913751808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433685.798248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433685.798249:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433685.798251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433685.798254:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751936:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433685.798263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433685.798268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433685/real 1789433685] req@ffff9811d2101180 x1876350913751936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433685.798274:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913751936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433685.798279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433685.798281:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433685.798282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913751936:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433686.694020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433687.717979:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433688.741976:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433689.765954:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433690.789982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433690.917925:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433690.917934:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433690.917939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433690.917947:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6060 00000100:00080000:2.0:1789433690.917954:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433690.917962:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433690.917991:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913752064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433690.918011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433690.918013:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433690.918014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433690.918023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6060 00000100:00080000:2.0:1789433690.918026:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433690.918034:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433690.918044:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433690.918057:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913752192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433690.918060:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913752064:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433690.918069:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433690.918074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433690.918075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433690.918081:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6060 00000100:00080000:2.0:1789433690.918084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433690.918087:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433690.918091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433690.918099:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913752320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433690.918114:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433690.918116:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433690.918117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433690.918120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6060 00000100:00080000:2.0:1789433690.918125:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433690.918127:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433690.918130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433690.918135:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789433690.918142:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913752448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433690.918177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433690/real 1789433690] req@ffff9811d2101880 x1876350913752064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433690.918195:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913752064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433690.918206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433690.918209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433690.918213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913752064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433690.918233:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433690.918251:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433690.918259:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433690/real 1789433690] req@ffff9811d2100e00 x1876350913752192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433690.918267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913752192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433690.918272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433690.918275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433690.918277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433690.918280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433690.918290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433690.918295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433690/real 1789433690] req@ffff9811d2101f80 x1876350913752320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433690.918301:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913752320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433690.918306:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433690.918307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433690.918309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433690.918312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752448:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433690.918321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433690.918329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433690/real 1789433690] req@ffff9811d2101180 x1876350913752448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433690.918340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913752448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433690.918347:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433690.918350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433690.918353:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752448:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433691.813987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433692.838002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433693.862012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433694.885992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433695.910012:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433696.038023:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433696.038030:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433696.038033:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433696.038040:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6065 00000100:00080000:3.0:1789433696.038047:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433696.038056:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433696.038089:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913752576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433696.038113:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433696.038117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433696.038119:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433696.038122:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6065 00000100:00080000:3.0:1789433696.038125:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433696.038128:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433696.038136:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433696.038146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913752704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433696.038154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433696.038158:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789433696.038159:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913752576:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433696.038159:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433696.038166:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6065 00000100:00080000:3.0:1789433696.038170:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433696.038173:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433696.038176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433696.038183:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913752832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433696.038190:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433696.038192:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433696.038193:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433696.038195:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6065 00000100:00080000:3.0:1789433696.038197:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433696.038199:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433696.038201:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433696.038209:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913752960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433696.038225:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433696.038265:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433696/real 1789433696] req@ffff9811d1bc4700 x1876350913752576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433696.038280:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913752576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433696.038288:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433696.038291:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433696.038293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913752576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433696.038310:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433696.038326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433696.038334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433696/real 1789433696] req@ffff9811d1bc5f80 x1876350913752704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433696.038341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913752704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433696.038345:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433696.038347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433696.038349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433696.038352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433696.038361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433696.038366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433696/real 1789433696] req@ffff9811d1bc5880 x1876350913752832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433696.038372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913752832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433696.038377:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433696.038378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433696.038380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433696.038383:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752960:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433696.038392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433696.038397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433696/real 1789433696] req@ffff9811d1bc5c00 x1876350913752960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433696.038403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913752960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433696.038407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433696.038409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433696.038411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913752960:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433696.934023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433697.957993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433698.981993:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433700.005949:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433701.030004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433701.157936:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433701.157946:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433701.157951:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433701.157960:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6070 00000100:00080000:1.0:1789433701.157971:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433701.157982:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433701.158016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913753088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433701.158042:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433701.158046:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433701.158048:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433701.158053:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6070 00000100:00080000:1.0:1789433701.158057:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433701.158063:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433701.158073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433701.158077:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913753088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433701.158089:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913753216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433701.158102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433701.158106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433701.158108:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433701.158111:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6070 00000100:00080000:1.0:1789433701.158118:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433701.158123:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433701.158128:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789433701.158129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433701.158145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913753344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433701.158156:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433701.158160:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433701.158162:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433701.158164:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433701/real 1789433701] req@ffff9811d0bde680 x1876350913753088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433701.158168:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6070 00000100:00080000:1.0:1789433701.158175:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789433701.158184:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913753088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433701.158184:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433701.158193:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789433701.158196:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433701.158199:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433701.158202:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913753088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433701.158211:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913753472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433701.158221:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433701.158237:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433701.158249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433701/real 1789433701] req@ffff9811d0bdf480 x1876350913753216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433701.158258:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913753216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433701.158263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433701.158266:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433701.158267:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433701.158270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433701.158280:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433701.158285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433701/real 1789433701] req@ffff9811d0bdfb80 x1876350913753344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433701.158291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913753344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433701.158295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433701.158297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433701.158298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433701.158306:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753472:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433701.158315:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433701.158321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433701/real 1789433701] req@ffff9811d0bdc700 x1876350913753472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433701.158329:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913753472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433701.158337:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433701.158340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433701.158342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753472:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433702.053969:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433703.077994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433704.101981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433705.126038:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433706.149982:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433706.278624:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433706.278637:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433706.278641:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433706.278651:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6075 00000100:00080000:3.0:1789433706.278660:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433706.278671:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433706.278707:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913753600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433706.278740:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433706.278744:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433706.278745:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433706.278749:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6075 00000100:00080000:3.0:1789433706.278753:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433706.278758:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433706.278768:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433706.278783:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913753600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433706.278784:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913753728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433706.278799:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433706.278802:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433706.278804:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433706.278807:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6075 00000100:00080000:3.0:1789433706.278811:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433706.278815:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433706.278820:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433706.278852:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913753856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433706.278863:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433706.278867:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433706.278869:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433706.278873:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6075 00000100:00080000:3.0:1789433706.278877:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433706.278882:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433706.278886:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433706.278892:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433706.278906:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913753984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433706.278921:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433706/real 1789433706] req@ffff9811d1bc4000 x1876350913753600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433706.278941:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913753600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433706.278951:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433706.278961:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433706.278965:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913753600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433706.278986:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433706.279011:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433706.279024:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433706/real 1789433706] req@ffff9811d1bc7800 x1876350913753728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433706.279036:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913753728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433706.279043:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433706.279046:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433706.279049:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433706.279054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433706.279073:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433706.279083:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433706/real 1789433706] req@ffff9811d1bc5180 x1876350913753856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433706.279093:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913753856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433706.279101:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433706.279104:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433706.279107:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433706.279113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753984:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433706.279141:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433706.279151:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433706/real 1789433706] req@ffff9811d1bc6a00 x1876350913753984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433706.279167:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913753984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433706.279174:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433706.279177:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433706.279180:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913753984:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433707.174031:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433708.198040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433709.221997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433710.245993:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433711.270033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433711.398051:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433711.398061:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433711.398065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433711.398075:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6080 00000100:00080000:1.0:1789433711.398083:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433711.398094:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433711.398128:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913754112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433711.398154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433711.398157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433711.398159:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433711.398162:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6080 00000100:00080000:1.0:1789433711.398165:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433711.398169:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433711.398178:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433711.398188:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913754240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433711.398197:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433711.398200:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433711.398202:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433711.398205:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6080 00000100:00080000:1.0:1789433711.398208:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433711.398212:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433711.398216:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433711.398224:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913754368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433711.398239:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433711.398242:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433711.398244:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433711.398247:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6080 00000100:00080000:1.0:1789433711.398249:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433711.398252:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433711.398256:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433711.398263:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913754496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433711.398304:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913754112:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433711.398360:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433711.398454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433711/real 1789433711] req@ffff9811d0bdea00 x1876350913754112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433711.398475:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913754112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433711.398485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433711.398489:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433711.398492:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913754112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433711.398499:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433711.398519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433711.398530:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433711/real 1789433711] req@ffff9811d0bded80 x1876350913754240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433711.398540:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913754240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433711.398547:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433711.398550:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433711.398553:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433711.398568:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433711.398585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433711.398593:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433711/real 1789433711] req@ffff9811d0bdc000 x1876350913754368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433711.398603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913754368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433711.398610:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433711.398613:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433711.398615:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433711.398621:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754496:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433711.398636:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433711.398645:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433711/real 1789433711] req@ffff9811d0bdd880 x1876350913754496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433711.398656:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913754496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433711.398664:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433711.398667:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433711.398670:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433712.294005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433713.318019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433714.341970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433715.365975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433716.390010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433716.517957:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433716.517968:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433716.517972:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433716.517982:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6086 00000100:00080000:3.0:1789433716.517992:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433716.518003:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433716.518039:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913754624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433716.518070:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433716.518073:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433716.518075:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433716.518079:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6086 00000100:00080000:3.0:1789433716.518083:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433716.518088:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433716.518097:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433716.518111:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913754752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433716.518119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913754624:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433716.518122:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433716.518126:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433716.518128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433716.518132:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6086 00000100:00080000:3.0:1789433716.518135:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433716.518139:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433716.518144:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433716.518156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913754880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433716.518164:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433716.518167:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433716.518169:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433716.518172:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6086 00000100:00080000:3.0:1789433716.518175:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433716.518179:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433716.518184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789433716.518186:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433716.518198:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913755008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433716.518224:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433716/real 1789433716] req@ffff9811d1bc7480 x1876350913754624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433716.518238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913754624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433716.518245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433716.518248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433716.518250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913754624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433716.518267:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433716.518282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433716.518289:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433716/real 1789433716] req@ffff9811d1bc6a00 x1876350913754752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433716.518297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913754752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433716.518303:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433716.518306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433716.518309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433716.518314:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433716.518334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433716.518345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433716/real 1789433716] req@ffff9811d1bc5180 x1876350913754880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433716.518357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913754880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433716.518365:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433716.518367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433716.518369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913754880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433716.518373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755008:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433716.518387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433716.518394:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433716/real 1789433716] req@ffff9811d1bc7800 x1876350913755008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433716.518404:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913755008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433716.518412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433716.518414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433716.518417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433717.414070:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433718.437998:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433719.462003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433720.486047:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433721.510056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433721.638019:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433721.638029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433721.638034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433721.638044:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6091 00000100:00080000:1.0:1789433721.638054:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433721.638065:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433721.638105:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913755136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433721.638128:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433721.638132:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433721.638134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433721.638141:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6091 00000100:00080000:1.0:1789433721.638145:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433721.638150:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433721.638160:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433721.638172:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913755264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433721.638183:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433721.638186:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433721.638188:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789433721.638193:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913755136:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433721.638195:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6091 00000100:00080000:1.0:1789433721.638201:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433721.638207:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433721.638211:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433721.638222:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913755392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433721.638232:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433721.638235:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433721.638237:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433721.638240:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6091 00000100:00080000:1.0:1789433721.638249:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433721.638253:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433721.638258:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433721.638261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433721.638273:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913755520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433721.638311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433721/real 1789433721] req@ffff9811d0bdf800 x1876350913755136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433721.638331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913755136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433721.638341:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433721.638344:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433721.638348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913755136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433721.638380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433721.638402:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433721.638414:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433721/real 1789433721] req@ffff9811d0bde300 x1876350913755264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433721.638425:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913755264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433721.638432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433721.638435:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433721.638438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433721.638442:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433721.638459:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433721.638468:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433721/real 1789433721] req@ffff9811d0bdf100 x1876350913755392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433721.638478:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913755392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433721.638504:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433721.638507:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433721.638510:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433721.638516:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755520:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433721.638535:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433721.638545:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433721/real 1789433721] req@ffff9811d0bdd880 x1876350913755520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433721.638555:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913755520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433721.638562:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433721.638566:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433721.638568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433722.534002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433723.558015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433724.582076:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433725.606064:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433726.630010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433726.757918:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433726.757927:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433726.757931:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433726.757951:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6096 00000100:00080000:1.0:1789433726.757962:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433726.757973:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433726.758005:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913755648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433726.758029:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433726.758032:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433726.758034:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433726.758037:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6096 00000100:00080000:1.0:1789433726.758040:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433726.758044:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433726.758052:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433726.758064:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913755776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433726.758073:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433726.758076:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433726.758077:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433726.758080:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6096 00000100:00080000:1.0:1789433726.758082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433726.758087:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433726.758091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913755648:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789433726.758093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433726.758104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913755904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433726.758118:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433726.758121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433726.758123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433726.758126:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6096 00000100:00080000:1.0:1789433726.758129:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433726.758141:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433726.758145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433726.758155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913756032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433726.758189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433726.758224:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433726.758247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433727.654009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433728.678012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433729.702022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789433729.830069:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789433729.830107:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789433729.830120:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433729.830154:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433729.830161:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433729.830168:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433729.830199:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433726/real 1789433729] req@ffff9811d0bdea00 x1876350913755904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433729.830224:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913755904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433729.830235:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433729.830240:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433729.830244:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433729.830252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433726/real 1789433729] req@ffff9811d0bdc700 x1876350913756032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433729.830265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913756032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433729.830271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433729.830274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433729.830276:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756032:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433729.830280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433726/real 1789433729] req@ffff9811d0bdc000 x1876350913755648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433729.830288:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913755648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433729.830295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433729.830298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433729.830301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913755648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433729.830306:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433726/real 1789433729] req@ffff9811d0bded80 x1876350913755776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433729.830316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913755776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433729.830323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433729.830331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433729.830335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913755776:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789433730.726009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433731.752079:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433731.878057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433731.878065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433731.878069:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433731.878079:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6101 00000100:00080000:3.0:1789433731.878088:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433731.878103:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433731.878142:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913756160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433731.878167:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433731.878170:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433731.878172:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433731.878177:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6101 00000100:00080000:3.0:1789433731.878180:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433731.878184:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433731.878194:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433731.878205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913756288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433731.878212:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433731.878215:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433731.878217:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433731.878220:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6101 00000100:00080000:3.0:1789433731.878222:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433731.878225:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433731.878229:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433731.878237:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913756416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433731.878245:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433731.878247:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433731.878248:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433731.878252:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6101 00000100:00080000:3.0:1789433731.878254:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433731.878258:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433731.878261:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433731.878282:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913756544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433731.878292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913756160:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433731.878362:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433731.878397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433731/real 1789433731] req@ffff9811d1bc4000 x1876350913756160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433731.878427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913756160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433731.878438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433731.878441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433731.878457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913756160:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433731.878463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433731.878491:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433731.878505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433731/real 1789433731] req@ffff9811d1bc5c00 x1876350913756288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433731.878518:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913756288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433731.878532:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433731.878536:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433731.878539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433731.878545:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433731.878563:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433731.878573:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433731/real 1789433731] req@ffff9811d1bc5880 x1876350913756416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433731.878593:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913756416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433731.878600:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433731.878603:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433731.878606:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433731.878627:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756544:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433731.878646:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433731.878657:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433731/real 1789433731] req@ffff9811d1bc5f80 x1876350913756544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433731.878668:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913756544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433731.878676:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433731.878679:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433731.878683:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433732.774127:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433733.797994:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433734.822088:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433735.846418:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433736.870356:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433736.997944:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433736.997954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433736.997958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433736.997967:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6106 00000100:00080000:3.0:1789433736.997976:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433736.997986:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433736.998018:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913756672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433736.998042:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433736.998046:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433736.998047:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433736.998051:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6106 00000100:00080000:3.0:1789433736.998053:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433736.998058:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433736.998068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433736.998079:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913756800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433736.998084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913756672:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433736.998096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433736.998099:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433736.998101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433736.998105:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6106 00000100:00080000:3.0:1789433736.998108:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433736.998112:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433736.998117:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433736.998127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913756928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433736.998136:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433736.998139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433736.998142:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433736.998145:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433736.998150:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6106 00000100:00080000:3.0:1789433736.998154:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433736.998159:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433736.998164:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433736.998172:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433736/real 1789433736] req@ffff9811d1bc4700 x1876350913756672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433736.998178:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913757056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433736.998201:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913756672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433736.998213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433736.998217:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433736.998221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913756672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433736.998248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433736.998270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433736.998282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433736/real 1789433736] req@ffff9811d1bc6680 x1876350913756800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433736.998292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913756800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433736.998297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433736.998299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433736.998301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433736.998305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433736.998318:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433736.998327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433736/real 1789433736] req@ffff9811d1bc5500 x1876350913756928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433736.998337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913756928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433736.998344:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433736.998347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433736.998351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913756928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433736.998356:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433736.998386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433736.998394:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433736/real 1789433736] req@ffff9811d1bc6300 x1876350913757056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433736.998401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913757056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433736.998406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433736.998408:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433736.998410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433737.893999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433738.918095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433739.941993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433740.966020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433741.990028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433742.118083:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433742.118091:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433742.118096:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433742.118105:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6111 00000100:00080000:3.0:1789433742.118114:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433742.118125:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433742.118158:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913757184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433742.118184:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433742.118188:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433742.118189:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433742.118193:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6111 00000100:00080000:3.0:1789433742.118196:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433742.118200:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433742.118209:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433742.118222:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913757312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433742.118231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913757184:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433742.118236:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433742.118241:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433742.118243:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433742.118246:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6111 00000100:00080000:3.0:1789433742.118249:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433742.118253:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433742.118258:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433742.118269:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913757440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433742.118279:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433742.118283:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433742.118285:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433742.118288:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6111 00000100:00080000:3.0:1789433742.118293:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433742.118304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433742.118305:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433742.118312:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433742.118324:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913757568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433742.118366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433742/real 1789433742] req@ffff9811d1bc7100 x1876350913757184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433742.118386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913757184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433742.118394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433742.118397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433742.118399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913757184:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433742.118416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433742.118432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433742.118440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433742/real 1789433742] req@ffff9811d1bc6300 x1876350913757312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433742.118447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913757312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433742.118452:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433742.118454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433742.118456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433742.118459:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433742.118467:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433742.118472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433742/real 1789433742] req@ffff9811d1bc5500 x1876350913757440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433742.118478:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913757440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433742.118487:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433742.118490:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433742.118493:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433742.118498:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757568:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433742.118528:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433742.118539:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433742/real 1789433742] req@ffff9811d1bc6680 x1876350913757568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433742.118549:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913757568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433742.118555:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433742.118557:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433742.118559:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433743.014021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433744.037982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433745.062004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433746.085993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433747.110008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433747.237940:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433747.237949:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433747.237953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433747.237972:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6116 00000100:00080000:3.0:1789433747.237981:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433747.237992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433747.238025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913757696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433747.238048:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433747.238051:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433747.238052:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433747.238056:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6116 00000100:00080000:3.0:1789433747.238059:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433747.238064:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433747.238073:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433747.238087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913757824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433747.238098:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433747.238106:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913757696:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433747.238106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433747.238110:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433747.238117:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6116 00000100:00080000:3.0:1789433747.238120:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433747.238124:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433747.238129:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433747.238141:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913757952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433747.238153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433747.238156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433747.238158:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433747.238162:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6116 00000100:00080000:3.0:1789433747.238173:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433747.238181:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433747.238181:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433747.238189:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433747.238206:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913758080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433747.238220:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433747/real 1789433747] req@ffff9811d1bc4700 x1876350913757696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433747.238238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913757696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433747.238249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433747.238253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433747.238257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913757696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433747.238286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433747.238322:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433747.238334:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433747/real 1789433747] req@ffff9811d1bc5f80 x1876350913757824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433747.238346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913757824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433747.238354:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433747.238358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433747.238361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433747.238367:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433747.238384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433747.238394:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433747/real 1789433747] req@ffff9811d1bc5880 x1876350913757952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433747.238405:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913757952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433747.238412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433747.238415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433747.238418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913757952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433747.238424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433747.238442:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433747.238452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433747/real 1789433747] req@ffff9811d1bc5c00 x1876350913758080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433747.238462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913758080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433747.238470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433747.238473:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433747.238476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433748.134024:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433749.158011:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433750.182036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433751.206014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433752.230056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433752.357961:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433752.357970:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433752.357974:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433752.357983:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6121 00000100:00080000:0.0:1789433752.357992:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433752.358001:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433752.358034:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913758208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433752.358067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433752.358071:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433752.358072:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433752.358076:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6121 00000100:00080000:0.0:1789433752.358079:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433752.358083:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433752.358091:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433752.358109:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913758336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433752.358125:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433752.358128:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433752.358130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433752.358134:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6121 00000100:00080000:0.0:1789433752.358138:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433752.358142:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433752.358147:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433752.358152:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913758208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433752.358173:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913758464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433752.358184:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433752.358187:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433752.358189:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433752.358193:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6121 00000100:00080000:0.0:1789433752.358196:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433752.358201:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433752.358205:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789433752.358212:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433752.358217:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913758592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433752.358253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433752/real 1789433752] req@ffff9811c8cd7b80 x1876350913758208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433752.358281:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913758208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433752.358297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433752.358301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433752.358305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913758208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433752.358312:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433752.358333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433752.358344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433752/real 1789433752] req@ffff9811c8cd6300 x1876350913758336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433752.358355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913758336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433752.358362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433752.358364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433752.358367:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433752.358387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433752.358402:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433752.358411:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433752/real 1789433752] req@ffff9811c8cd4000 x1876350913758464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433752.358420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913758464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433752.358427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433752.358429:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433752.358432:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433752.358438:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758592:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433752.358452:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433752.358460:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433752/real 1789433752] req@ffff9811c8cd6a00 x1876350913758592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433752.358474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913758592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433752.358482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433752.358485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433752.358487:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433753.254009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433754.277996:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433755.302003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433756.326021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433757.350159:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433757.477994:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433757.478008:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433757.478014:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433757.478028:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6127 00000100:00080000:0.0:1789433757.478038:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433757.478053:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433757.478109:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913758720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433757.478136:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433757.478140:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433757.478141:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433757.478146:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6127 00000100:00080000:0.0:1789433757.478150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433757.478156:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433757.478168:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433757.478184:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913758848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433757.478184:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913758720:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433757.478209:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433757.478212:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433757.478214:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433757.478229:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6127 00000100:00080000:0.0:1789433757.478235:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433757.478240:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433757.478246:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433757.478259:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913758976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433757.478276:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789433757.478284:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789433757.478286:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433757.478289:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433757.478295:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6127 00000100:00080000:0.0:1789433757.478300:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433757.478308:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433757.478312:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433757.478326:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913759104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433757.478336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433757/real 1789433757] req@ffff9811c8cd5180 x1876350913758720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433757.478359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913758720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433757.478368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433757.478372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433757.478376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913758720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433757.478404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433757.478429:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433757.478443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433757/real 1789433757] req@ffff9811c8cd7800 x1876350913758848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433757.478454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913758848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433757.478463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433757.478466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433757.478470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433757.478476:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433757.478495:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433757.478505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433757/real 1789433757] req@ffff9811c8cd7480 x1876350913758976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433757.478516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913758976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433757.478521:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433757.478530:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433757.478532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913758976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433757.478538:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433757.478552:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433757.478558:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433757/real 1789433757] req@ffff9811c8cd7100 x1876350913759104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433757.478565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913759104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433757.478574:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433757.478576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433757.478578:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433758.373983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433759.398015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433760.422003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433761.446058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433762.470049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433762.598234:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433762.598244:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433762.598248:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433762.598258:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6132 00000100:00080000:0.0:1789433762.598267:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433762.598278:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433762.598314:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913759232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433762.598350:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433762.598353:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433762.598355:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433762.598360:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6132 00000100:00080000:0.0:1789433762.598367:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433762.598387:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433762.598395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913759232:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789433762.598397:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433762.598415:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913759360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433762.598433:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433762.598437:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433762.598439:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433762.598443:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6132 00000100:00080000:0.0:1789433762.598447:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433762.598452:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433762.598458:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789433762.598459:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433762.598482:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913759488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433762.598493:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789433762.598498:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433762/real 1789433762] req@ffff9811c8cd5c00 x1876350913759232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433762.598499:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433762.598502:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433762.598511:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6132 00000100:00080000:0.0:1789433762.598518:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433762.598525:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433762.598528:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913759232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789433762.598532:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789433762.598545:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1789433762.598550:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913759616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433762.598551:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433762.598558:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913759232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433762.598585:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433762.598603:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433762.598613:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433762/real 1789433762] req@ffff9811c8cd4700 x1876350913759360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433762.598624:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913759360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433762.598632:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433762.598635:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433762.598638:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433762.598644:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433762.598656:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433762.598664:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433762/real 1789433762] req@ffff9811c8cd6680 x1876350913759488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433762.598674:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913759488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433762.598689:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433762.598692:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433762.598695:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433762.598700:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433762.598714:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433762.598722:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433762/real 1789433762] req@ffff9811c8cd7100 x1876350913759616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433762.598732:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913759616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433762.598740:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433762.598743:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433762.598746:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433763.494001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433764.518002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433765.542082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433766.566008:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433767.590039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433767.718286:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433767.718296:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433767.718299:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433767.718309:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6137 00000100:00080000:1.0:1789433767.718318:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433767.718330:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433767.718363:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913759744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433767.718397:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433767.718406:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433767.718408:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433767.718413:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6137 00000100:00080000:1.0:1789433767.718418:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433767.718423:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433767.718434:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433767.718438:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913759744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433767.718459:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913759872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433767.718472:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433767.718475:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433767.718477:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433767.718481:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6137 00000100:00080000:1.0:1789433767.718485:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433767.718495:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789433767.718501:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789433767.718504:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433767.718519:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913760000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433767.718530:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433767.718533:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433767.718535:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433767.718540:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6137 00000100:00100000:3.0:1789433767.718541:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433767/real 1789433767] req@ffff9811d0bdfb80 x1876350913759744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433767.718545:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433767.718554:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433767.718560:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433767.718563:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913759744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433767.718572:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789433767.718576:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913760128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433767.718576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433767.718582:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913759744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433767.718605:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433767.718622:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433767.718630:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433767/real 1789433767] req@ffff9811d0bdf480 x1876350913759872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433767.718638:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913759872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433767.718642:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433767.718644:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433767.718646:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913759872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433767.718649:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433767.718659:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433767.718667:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433767/real 1789433767] req@ffff9811d0bde680 x1876350913760000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433767.718678:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913760000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433767.718690:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433767.718693:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433767.718696:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433767.718702:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760128:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433767.718719:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433767.718726:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433767/real 1789433767] req@ffff9811d0bdd500 x1876350913760128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433767.718733:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913760128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433767.718738:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433767.718740:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433767.718742:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433768.614020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433769.638015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433770.662083:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433771.686061:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433772.709998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433772.837984:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433772.837991:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433772.837995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433772.838003:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6142 00000100:00080000:1.0:1789433772.838010:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433772.838017:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433772.838044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913760256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433772.838063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433772.838065:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433772.838067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433772.838070:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6142 00000100:00080000:1.0:1789433772.838072:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433772.838074:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433772.838080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433772.838088:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913760384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433772.838095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433772.838097:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433772.838098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433772.838100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6142 00000100:00080000:1.0:1789433772.838101:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433772.838113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913760256:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433772.838119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433772.838124:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433772.838134:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913760512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433772.838146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433772.838148:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433772.838150:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433772.838152:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6142 00000100:00080000:1.0:1789433772.838155:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433772.838159:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433772.838162:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433772.838169:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913760640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433772.838176:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433772.838211:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433772/real 1789433772] req@ffff9811d0bddc00 x1876350913760256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433772.838229:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913760256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433772.838237:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433772.838240:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433772.838243:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913760256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433772.838248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433772.838264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433772.838275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433772/real 1789433772] req@ffff9811d0bdc380 x1876350913760384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433772.838297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913760384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433772.838304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433772.838308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433772.838311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433772.838332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433772.838352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433772.838364:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433772/real 1789433772] req@ffff9811d0bdd180 x1876350913760512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433772.838375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913760512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433772.838380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433772.838383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433772.838385:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433772.838388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760640:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433772.838411:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433772.838421:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433772/real 1789433772] req@ffff9811d0bdd500 x1876350913760640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433772.838433:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913760640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433772.838441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433772.838443:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433772.838446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433773.734052:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433774.758017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433775.782023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433776.805986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433777.830242:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433777.958116:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433777.958125:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433777.958129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433777.958139:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6147 00000100:00080000:3.0:1789433777.958147:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433777.958157:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433777.958188:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913760768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433777.958223:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433777.958226:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433777.958228:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433777.958233:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6147 00000100:00080000:3.0:1789433777.958236:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433777.958241:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433777.958250:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433777.958262:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913760896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433777.958276:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433777.958279:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433777.958281:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433777.958284:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6147 00000100:00080000:3.0:1789433777.958287:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433777.958291:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433777.958295:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433777.958305:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913761024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433777.958313:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433777.958316:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433777.958318:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433777.958321:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6147 00000100:00080000:3.0:1789433777.958324:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433777.958328:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433777.958332:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433777.958343:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913761152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433777.958417:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913760768:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433777.958469:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433777.958504:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433777/real 1789433777] req@ffff9811d1bc4000 x1876350913760768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433777.958516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913760768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433777.958522:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433777.958524:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433777.958527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913760768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433777.958532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433777.958544:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433777.958551:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433777/real 1789433777] req@ffff9811d1bc7800 x1876350913760896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433777.958558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913760896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433777.958562:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433777.958564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433777.958566:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913760896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433777.958569:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433777.958578:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433777.958583:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433777/real 1789433777] req@ffff9811d1bc5180 x1876350913761024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433777.958589:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913761024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433777.958594:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433777.958595:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433777.958597:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433777.958599:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761152:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433777.958608:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433777.958612:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433777/real 1789433777] req@ffff9811d1bc6a00 x1876350913761152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433777.958618:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913761152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433777.958623:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433777.958625:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433777.958626:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433778.854208:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433779.878000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433780.902194:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433781.926003:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433782.950023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433783.078408:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433783.078418:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433783.078422:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433783.078433:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6152 00000100:00080000:3.0:1789433783.078442:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433783.078454:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433783.078491:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913761280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433783.078519:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433783.078523:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433783.078525:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433783.078529:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6152 00000100:00080000:3.0:1789433783.078532:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433783.078537:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433783.078546:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433783.078561:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913761408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433783.078572:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913761280:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433783.078574:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433783.078582:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433783.078584:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433783.078588:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6152 00000100:00080000:3.0:1789433783.078592:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433783.078597:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433783.078601:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433783.078613:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913761536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433783.078622:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433783.078626:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433783.078628:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789433783.078639:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433783.078640:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6152 00000100:00080000:3.0:1789433783.078646:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433783.078652:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433783.078657:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433783.078670:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913761664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433783.078681:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433783/real 1789433783] req@ffff9811d1bc7480 x1876350913761280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433783.078701:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913761280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433783.078712:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433783.078716:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433783.078720:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913761280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433783.078741:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433783.078761:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433783.078771:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433783/real 1789433783] req@ffff9811d1bc6a00 x1876350913761408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433783.078783:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913761408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433783.078791:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433783.078794:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433783.078798:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433783.078803:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433783.078818:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433783.078826:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433783/real 1789433783] req@ffff9811d1bc5180 x1876350913761536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433783.078870:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913761536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433783.078885:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433783.078888:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433783.078891:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433783.078910:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433783.078932:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433783.078943:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433783/real 1789433783] req@ffff9811d1bc7800 x1876350913761664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433783.078954:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913761664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433783.078961:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433783.078964:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433783.078967:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433783.973986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433784.998011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433786.022042:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433787.046012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433788.070095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433788.197944:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433788.197953:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433788.197957:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433788.197966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6157 00000100:00080000:3.0:1789433788.197974:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433788.197984:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433788.198037:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913761792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433788.198060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433788.198064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433788.198066:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433788.198069:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6157 00000100:00080000:3.0:1789433788.198073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433788.198077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433788.198086:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433788.198098:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913761920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433788.198108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913761792:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433788.198110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433788.198117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433788.198120:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433788.198124:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6157 00000100:00080000:3.0:1789433788.198127:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433788.198131:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433788.198135:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433788.198146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913762048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433788.198156:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433788.198159:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433788.198161:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433788.198164:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6157 00000100:00080000:3.0:1789433788.198168:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433788.198171:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433788.198175:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433788.198185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433788.198187:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913762176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433788.198239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433788/real 1789433788] req@ffff9811d1bc4000 x1876350913761792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433788.198262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913761792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433788.198273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433788.198278:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433788.198283:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913761792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433788.198305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433788.198327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433788.198339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433788/real 1789433788] req@ffff9811d1bc5c00 x1876350913761920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433788.198351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913761920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433788.198359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433788.198363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433788.198366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913761920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433788.198372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433788.198388:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433788.198398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433788/real 1789433788] req@ffff9811d1bc5880 x1876350913762048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433788.198414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913762048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433788.198423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433788.198426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433788.198429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433788.198434:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433788.198453:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433788.198463:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433788/real 1789433788] req@ffff9811d1bc5f80 x1876350913762176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433788.198474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913762176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433788.198482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433788.198486:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433788.198489:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433789.094074:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433790.118030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433791.141997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433792.166054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433793.190137:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433793.317979:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433793.317989:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433793.317993:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433793.318004:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6162 00000100:00080000:3.0:1789433793.318013:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433793.318026:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433793.318061:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913762304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433793.318086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433793.318089:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433793.318090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433793.318093:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6162 00000100:00080000:3.0:1789433793.318096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433793.318099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433793.318108:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433793.318122:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913762432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433793.318124:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913762304:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433793.318140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433793.318143:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433793.318145:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433793.318149:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6162 00000100:00080000:3.0:1789433793.318152:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433793.318158:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433793.318163:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433793.318172:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913762560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433793.318179:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433793.318182:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433793.318183:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433793.318185:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6162 00000100:00080000:3.0:1789433793.318194:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433793.318197:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433793.318201:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433793.318211:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913762688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433793.318249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433793.318292:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433793.318319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433794.214089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433795.237969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433796.261992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789433796.390079:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:2.0:1789433796.390117:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789433796.390133:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433796.390172:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433796.390181:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789433796.390190:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433796.390212:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433793/real 1789433796] req@ffff9811d1bc6680 x1876350913762432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433796.390237:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913762432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433796.390248:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433796.390253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433796.390256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433796.390264:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433793/real 1789433796] req@ffff9811d1bc5500 x1876350913762560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433796.390275:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913762560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433796.390282:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433796.390285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433796.390288:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433796.390293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433793/real 1789433796] req@ffff9811d1bc6300 x1876350913762688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433796.390303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913762688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433796.390311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433796.390314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433796.390316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762688:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433796.390321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433793/real 1789433796] req@ffff9811d1bc4700 x1876350913762304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433796.390337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913762304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433796.390344:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433796.390346:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433796.390349:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913762304:192.168.203.128@tcp:250:kworker.0 00010000:00010000:0.0:1789433797.286186:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433798.310017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433798.437950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433798.437960:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433798.437964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433798.437974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6168 00000100:00080000:3.0:1789433798.437982:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433798.437992:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433798.438024:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913762816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433798.438054:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433798.438058:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433798.438060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433798.438064:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6168 00000100:00080000:3.0:1789433798.438067:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433798.438072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433798.438082:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433798.438096:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913762944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433798.438104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913762816:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433798.438110:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433798.438115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433798.438116:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433798.438120:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6168 00000100:00080000:3.0:1789433798.438124:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433798.438128:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433798.438133:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433798.438146:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913763072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433798.438156:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433798.438161:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433798.438162:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433798.438167:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433798.438172:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6168 00000100:00080000:3.0:1789433798.438177:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433798.438183:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433798.438188:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433798.438200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433798/real 1789433798] req@ffff9811d1bc7100 x1876350913762816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433798.438204:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913763200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433798.438233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913762816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433798.438245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433798.438249:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433798.438252:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913762816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433798.438288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433798.438311:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433798.438324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433798/real 1789433798] req@ffff9811d1bc4700 x1876350913762944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433798.438335:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913762944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433798.438344:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433798.438347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433798.438351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913762944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433798.438357:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433798.438375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433798.438386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433798/real 1789433798] req@ffff9811d1bc6300 x1876350913763072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433798.438396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913763072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433798.438404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433798.438406:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433798.438409:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433798.438414:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433798.438433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433798.438444:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433798/real 1789433798] req@ffff9811d1bc5500 x1876350913763200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433798.438455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913763200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433798.438463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433798.438466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433798.438469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433799.333992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433800.358048:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433801.382049:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433802.406125:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433803.430289:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433803.558490:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433803.558500:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433803.558504:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433803.558514:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6173 00000100:00080000:2.0:1789433803.558521:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433803.558532:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433803.558566:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913763328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433803.558591:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433803.558594:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433803.558596:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433803.558599:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6173 00000100:00080000:2.0:1789433803.558603:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433803.558607:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433803.558615:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433803.558626:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913763456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433803.558635:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433803.558639:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433803.558641:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433803.558644:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6173 00000100:00080000:2.0:1789433803.558647:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433803.558651:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433803.558655:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433803.558663:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913763584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433803.558673:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433803.558676:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433803.558678:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433803.558689:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6173 00000100:00080000:2.0:1789433803.558692:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433803.558696:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433803.558698:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913763328:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789433803.558701:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433803.558715:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913763712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433803.558810:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433803.558921:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433803/real 1789433803] req@ffff9811d2101500 x1876350913763328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433803.558964:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913763328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433803.558976:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433803.558981:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433803.558986:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913763328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433803.558995:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433803.559029:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433803.559041:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433803/real 1789433803] req@ffff9811d2101c00 x1876350913763456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433803.559053:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913763456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433803.559061:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433803.559065:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433803.559068:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433803.559096:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433803.559117:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433803.559127:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433803/real 1789433803] req@ffff9811d2103b80 x1876350913763584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433803.559140:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913763584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433803.559154:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433803.559157:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433803.559160:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433803.559167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433803.559186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433803.559198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433803/real 1789433803] req@ffff9811d2102a00 x1876350913763712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433803.559209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913763712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433803.559218:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433803.559222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433803.559225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433804.454012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433805.478078:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433806.502074:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433807.526087:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433808.550255:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433808.678188:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433808.678197:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433808.678200:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433808.678210:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6178 00000100:00080000:1.0:1789433808.678221:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433808.678232:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433808.678263:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913763840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433808.678291:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433808.678294:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433808.678296:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433808.678300:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6178 00000100:00080000:1.0:1789433808.678303:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433808.678307:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433808.678315:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433808.678328:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913763968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433808.678337:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433808.678340:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433808.678341:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433808.678344:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6178 00000100:00080000:1.0:1789433808.678347:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433808.678350:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433808.678354:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433808.678361:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913764096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433808.678371:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433808.678374:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433808.678375:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433808.678378:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6178 00000100:00080000:1.0:1789433808.678381:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433808.678384:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433808.678388:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433808.678396:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913764224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433808.678468:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913763840:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433808.678519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433808.678563:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433808/real 1789433808] req@ffff9811d0bde680 x1876350913763840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433808.678578:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913763840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433808.678588:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433808.678592:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433808.678595:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913763840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433808.678601:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433808.678619:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433808.678639:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433808/real 1789433808] req@ffff9811d0bdf480 x1876350913763968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433808.678650:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913763968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433808.678658:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433808.678660:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433808.678663:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913763968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433808.678668:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433808.678683:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433808.678701:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433808/real 1789433808] req@ffff9811d0bdfb80 x1876350913764096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433808.678712:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913764096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433808.678719:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433808.678722:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433808.678725:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433808.678730:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764224:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433808.678746:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433808.678754:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433808/real 1789433808] req@ffff9811d0bded80 x1876350913764224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433808.678765:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913764224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433808.678772:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433808.678775:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433808.678778:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433809.574018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433810.598104:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433811.622128:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433812.647238:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433813.670007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433813.798255:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433813.798265:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433813.798268:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433813.798278:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6183 00000100:00080000:0.0:1789433813.798286:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433813.798295:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433813.798335:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913764352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433813.798367:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433813.798371:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433813.798373:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433813.798376:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6183 00000100:00080000:0.0:1789433813.798380:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433813.798385:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433813.798393:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433813.798406:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913764480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433813.798414:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433813.798417:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433813.798419:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433813.798421:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6183 00000100:00080000:0.0:1789433813.798424:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433813.798427:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433813.798430:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433813.798438:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913764608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433813.798446:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433813.798449:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433813.798451:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433813.798454:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6183 00000100:00080000:0.0:1789433813.798456:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433813.798460:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433813.798463:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433813.798470:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913764736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433813.798543:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913764352:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433813.798595:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433813.798626:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433813/real 1789433813] req@ffff9811c8cd7480 x1876350913764352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433813.798659:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913764352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433813.798667:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433813.798671:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433813.798674:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913764352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433813.798680:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433813.798697:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433813.798707:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433813/real 1789433813] req@ffff9811c8cd7800 x1876350913764480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433813.798717:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913764480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433813.798724:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433813.798727:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433813.798730:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433813.798735:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433813.798749:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433813.798758:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433813/real 1789433813] req@ffff9811c8cd5180 x1876350913764608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433813.798768:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913764608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433813.798779:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433813.798781:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433813.798784:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433813.798789:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433813.798820:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433813.798830:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433813/real 1789433813] req@ffff9811c8cd6a00 x1876350913764736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433813.798863:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913764736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433813.798870:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433813.798873:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433813.798876:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433814.694018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433815.718028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433816.742060:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433817.766383:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433818.790948:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433818.918743:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433818.918753:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433818.918756:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433818.918777:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6188 00000100:00080000:2.0:1789433818.918787:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433818.918799:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433818.918866:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913764864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433818.918898:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433818.918902:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433818.918904:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433818.918909:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6188 00000100:00080000:2.0:1789433818.918912:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433818.918926:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433818.918936:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433818.918948:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913764992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433818.918957:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433818.918960:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433818.918962:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433818.918965:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6188 00000100:00080000:2.0:1789433818.918969:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433818.918973:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433818.918978:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433818.918988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913765120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433818.918997:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433818.918999:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433818.919001:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433818.919004:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6188 00000100:00080000:2.0:1789433818.919006:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433818.919011:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433818.919015:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433818.919024:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913765248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433818.919373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913764864:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433818.919425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433818.919458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433818/real 1789433818] req@ffff9811d2100000 x1876350913764864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433818.919475:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913764864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433818.919485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433818.919488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433818.919493:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913764864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433818.919499:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433818.919520:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433818.919531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433818/real 1789433818] req@ffff9811d2103800 x1876350913764992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433818.919542:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913764992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433818.919549:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433818.919552:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433818.919555:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913764992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433818.919560:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433818.919576:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433818.919585:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433818/real 1789433818] req@ffff9811d2102d80 x1876350913765120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433818.919594:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913765120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433818.919601:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433818.919604:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433818.919607:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433818.919612:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765248:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433818.919630:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433818.919639:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433818/real 1789433818] req@ffff9811d2102a00 x1876350913765248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433818.919650:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913765248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433818.919657:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433818.919659:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433818.919663:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433819.813978:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433820.837989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433821.862156:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433822.886014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433823.910001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433824.038017:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433824.038027:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433824.038032:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433824.038042:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6193 00000100:00080000:1.0:1789433824.038050:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433824.038061:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433824.038095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913765376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433824.038119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433824.038123:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433824.038124:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433824.038128:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6193 00000100:00080000:1.0:1789433824.038131:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433824.038146:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433824.038154:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913765376:192.168.203.128@tcp:250:kworker.0 00000008:00100000:1.0:1789433824.038156:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433824.038173:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913765504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433824.038183:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433824.038186:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433824.038188:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433824.038191:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6193 00000100:00080000:1.0:1789433824.038194:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433824.038199:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433824.038203:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789433824.038208:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433824.038216:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913765632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433824.038233:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433824.038236:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433824.038237:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433824.038241:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6193 00000100:00080000:1.0:1789433824.038245:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433824.038249:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433824.038254:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433824/real 1789433824] req@ffff9811d0bdc000 x1876350913765376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789433824.038263:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433824.038273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913765376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433824.038278:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913765760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433824.038296:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433824.038302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433824.038307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913765376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433824.038330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433824.038353:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433824.038368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433824/real 1789433824] req@ffff9811d0bdc700 x1876350913765504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433824.038387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913765504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433824.038395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433824.038399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433824.038402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433824.038408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433824.038422:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433824.038430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433824/real 1789433824] req@ffff9811d0bdea00 x1876350913765632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433824.038440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913765632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433824.038448:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433824.038452:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433824.038454:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433824.038460:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433824.038478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433824.038486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433824/real 1789433824] req@ffff9811d0bdd880 x1876350913765760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433824.038493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913765760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433824.038502:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433824.038504:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433824.038506:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913765760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433824.934057:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433825.957989:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433826.982042:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433828.006043:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433829.030106:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433829.157950:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433829.157960:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433829.157964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433829.157974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6198 00000100:00080000:1.0:1789433829.157982:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433829.157993:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433829.158027:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913765888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433829.158053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433829.158057:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433829.158059:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433829.158063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6198 00000100:00080000:1.0:1789433829.158066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433829.158076:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433829.158087:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433829.158100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913765888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433829.158104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913766016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433829.158117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433829.158121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433829.158123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433829.158127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6198 00000100:00080000:1.0:1789433829.158131:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433829.158137:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433829.158142:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433829.158153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913766144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433829.158160:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433829.158170:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433829.158174:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433829.158177:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433829.158181:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6198 00000100:00080000:1.0:1789433829.158184:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789433829.158193:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433829/real 1789433829] req@ffff9811d0bdf100 x1876350913765888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789433829.158202:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433829.158208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913765888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789433829.158209:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:3.0:1789433829.158222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433829.158225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433829.158231:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913765888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433829.158232:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913766272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433829.158250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433829.158267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433829.158274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433829/real 1789433829] req@ffff9811d0bde300 x1876350913766016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433829.158283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913766016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433829.158291:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433829.158294:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433829.158298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433829.158303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433829.158320:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433829.158330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433829/real 1789433829] req@ffff9811d0bdf800 x1876350913766144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433829.158342:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913766144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433829.158350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433829.158353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433829.158356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433829.158361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433829.158377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433829.158386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433829/real 1789433829] req@ffff9811d0bdd880 x1876350913766272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433829.158395:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913766272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433829.158403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433829.158405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433829.158408:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433830.054054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433831.078120:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433832.102413:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433833.126039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433834.150022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433834.278529:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433834.278538:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433834.278542:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433834.278552:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6203 00000100:00080000:1.0:1789433834.278561:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433834.278570:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433834.278609:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913766400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433834.278644:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433834.278648:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433834.278650:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433834.278665:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6203 00000100:00080000:1.0:1789433834.278669:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433834.278674:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433834.278683:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433834.278695:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913766528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433834.278700:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913766400:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433834.278704:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433834.278708:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433834.278709:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433834.278712:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6203 00000100:00080000:1.0:1789433834.278714:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433834.278717:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433834.278720:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433834.278727:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913766656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433834.278738:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433834.278740:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433834.278741:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433834.278744:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6203 00000100:00080000:1.0:1789433834.278745:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433834.278748:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433834.278751:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433834.278757:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913766784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433834.278764:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433834.278805:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433834/real 1789433834] req@ffff9811d0bdea00 x1876350913766400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433834.278823:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913766400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433834.278857:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433834.278862:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433834.278866:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913766400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433834.278888:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433834.278911:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433834.278923:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433834/real 1789433834] req@ffff9811d0bdc700 x1876350913766528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433834.278935:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913766528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433834.278943:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433834.278946:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433834.278949:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433834.278955:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433834.278974:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433834.278982:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433834/real 1789433834] req@ffff9811d0bdc000 x1876350913766656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433834.278993:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913766656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433834.278998:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433834.279000:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433834.279002:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433834.279005:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766784:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433834.279016:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433834.279021:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433834/real 1789433834] req@ffff9811d0bded80 x1876350913766784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433834.279028:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913766784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433834.279032:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433834.279034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433834.279035:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913766784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433835.174143:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433836.198040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433837.222313:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433838.246084:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433839.270011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433839.397944:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433839.397954:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433839.397958:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433839.397967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6208 00000100:00080000:2.0:1789433839.397976:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433839.397987:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433839.398019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913766912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433839.398045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433839.398049:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433839.398051:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433839.398056:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6208 00000100:00080000:2.0:1789433839.398060:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433839.398066:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433839.398076:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433839.398080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913766912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789433839.398091:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913767040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433839.398103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433839.398107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433839.398110:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433839.398114:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6208 00000100:00080000:2.0:1789433839.398128:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433839.398133:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789433839.398139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789433839.398140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433839.398156:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913767168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433839.398167:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433839.398170:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433839.398172:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789433839.398177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433839/real 1789433839] req@ffff9811d2103b80 x1876350913766912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789433839.398179:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6208 00000100:00080000:2.0:1789433839.398185:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433839.398190:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433839.398195:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433839.398206:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913767296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433839.398307:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913766912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433839.398318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433839.398322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433839.398326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913766912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433839.398347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433839.398369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433839.398387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433839/real 1789433839] req@ffff9811d2101c00 x1876350913767040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433839.398403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913767040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433839.398411:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433839.398419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433839.398424:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433839.398430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433839.398463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433839.398478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433839/real 1789433839] req@ffff9811d2101500 x1876350913767168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433839.398492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913767168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433839.398499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433839.398502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433839.398508:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433839.398514:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433839.398535:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433839.398550:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433839/real 1789433839] req@ffff9811d2101180 x1876350913767296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433839.398564:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913767296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433839.398573:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433839.398579:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433839.398583:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433840.294109:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433841.318000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433842.341982:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433843.366015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433844.390158:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433844.518041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433844.518051:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433844.518055:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433844.518064:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6214 00000100:00080000:3.0:1789433844.518072:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433844.518083:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433844.518118:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913767424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433844.518153:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433844.518157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433844.518159:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433844.518163:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6214 00000100:00080000:3.0:1789433844.518167:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433844.518172:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433844.518181:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433844.518193:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913767552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433844.518203:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433844.518206:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433844.518207:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433844.518211:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6214 00000100:00080000:3.0:1789433844.518213:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433844.518217:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433844.518220:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433844.518228:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913767680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433844.518237:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433844.518240:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433844.518242:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433844.518245:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6214 00000100:00080000:3.0:1789433844.518248:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433844.518255:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433844.518259:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433844.518265:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913767808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433844.518444:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913767424:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433844.518500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433844.518545:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433844/real 1789433844] req@ffff9811d1bc6680 x1876350913767424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433844.518560:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913767424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433844.518567:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433844.518569:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433844.518572:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913767424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433844.518588:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433844.518616:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433844.518630:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433844/real 1789433844] req@ffff9811d1bc5f80 x1876350913767552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433844.518642:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913767552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433844.518650:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433844.518654:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433844.518657:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433844.518663:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433844.518683:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433844.518692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433844/real 1789433844] req@ffff9811d1bc5880 x1876350913767680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433844.518703:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913767680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433844.518708:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433844.518710:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433844.518712:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433844.518715:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767808:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433844.518728:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433844.518734:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433844/real 1789433844] req@ffff9811d1bc5c00 x1876350913767808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433844.518741:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913767808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433844.518745:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433844.518747:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433844.518749:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913767808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433845.414007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433846.438017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433847.461984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433848.485999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433849.510642:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433849.637921:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433849.637931:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433849.637935:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433849.637945:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6219 00000100:00080000:3.0:1789433849.637954:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433849.637965:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433849.637997:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913767936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433849.638021:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433849.638024:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433849.638025:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433849.638030:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6219 00000100:00080000:3.0:1789433849.638033:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433849.638036:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433849.638043:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433849.638052:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913768064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433849.638058:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433849.638066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913767936:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433849.638067:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433849.638070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433849.638077:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6219 00000100:00080000:3.0:1789433849.638079:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433849.638081:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433849.638085:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433849.638091:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913768192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433849.638097:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433849.638099:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433849.638105:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433849.638109:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6219 00000100:00080000:3.0:1789433849.638113:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433849.638117:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433849.638122:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433849.638141:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433849.638150:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913768320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433849.638182:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433849/real 1789433849] req@ffff9811d1bc4000 x1876350913767936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433849.638199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913767936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433849.638224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433849.638227:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433849.638230:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913767936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433849.638251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433849.638271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433849.638282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433849/real 1789433849] req@ffff9811d1bc7800 x1876350913768064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433849.638292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913768064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433849.638299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433849.638302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433849.638305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433849.638311:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433849.638327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433849.638337:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433849/real 1789433849] req@ffff9811d1bc5180 x1876350913768192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433849.638359:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913768192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433849.638367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433849.638371:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433849.638374:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433849.638381:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433849.638408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433849.638420:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433849/real 1789433849] req@ffff9811d1bc6a00 x1876350913768320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433849.638432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913768320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433849.638440:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433849.638443:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433849.638446:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433850.534077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433851.558001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433852.582559:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433853.606570:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433854.630028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433854.759475:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433854.759486:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433854.759490:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433854.759511:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6224 00000100:00080000:2.0:1789433854.759522:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433854.759533:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433854.759573:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913768448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433854.759614:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433854.759625:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433854.759627:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433854.759632:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6224 00000100:00080000:2.0:1789433854.759635:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433854.759640:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433854.759650:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433854.759662:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913768576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433854.759671:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433854.759674:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433854.759676:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433854.759679:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6224 00000100:00080000:2.0:1789433854.759682:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433854.759686:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433854.759690:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433854.759699:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913768704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433854.759719:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433854.759722:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433854.759724:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433854.759727:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6224 00000100:00080000:2.0:1789433854.759731:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433854.759735:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433854.759739:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433854.759758:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913768832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433854.759858:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913768448:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433854.759912:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433854.759953:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433854/real 1789433854] req@ffff9811d2101f80 x1876350913768448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433854.759971:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913768448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433854.759981:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433854.759985:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433854.759988:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913768448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433854.759995:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433854.760015:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433854.760026:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433854/real 1789433854] req@ffff9811d2100e00 x1876350913768576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433854.760037:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913768576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433854.760044:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433854.760046:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433854.760049:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433854.760054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433854.760068:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433854.760078:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433854/real 1789433854] req@ffff9811d2101880 x1876350913768704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433854.760087:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913768704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433854.760093:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433854.760096:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433854.760099:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433854.760103:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433854.760118:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433854.760126:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433854/real 1789433854] req@ffff9811d2101180 x1876350913768832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433854.760135:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913768832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433854.760148:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433854.760150:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433854.760153:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913768832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433855.654168:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433856.678016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433857.702014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433858.726146:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433859.750025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433859.878011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433859.878021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433859.878025:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433859.878035:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6229 00000100:00080000:2.0:1789433859.878044:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433859.878056:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433859.878093:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913768960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433859.878123:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433859.878126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433859.878128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433859.878133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6229 00000100:00080000:2.0:1789433859.878136:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433859.878140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433859.878149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433859.878162:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913769088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433859.878171:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913768960:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789433859.878176:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433859.878180:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433859.878182:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433859.878186:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6229 00000100:00080000:2.0:1789433859.878189:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433859.878192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433859.878201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433859.878214:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913769216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433859.878224:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433859.878227:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433859.878229:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433859.878232:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6229 00000100:00080000:2.0:1789433859.878235:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433859.878239:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433859.878244:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789433859.878253:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913769344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433859.878290:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433859.878315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433859.878341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433860.774067:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433861.797980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433862.821986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433862.950233:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433862.950267:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433862.950278:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433862.950310:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433862.950316:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433862.950321:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433862.951551:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433859/real 1789433862] req@ffff9811d2101c00 x1876350913769088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433862.951580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913769088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433862.951591:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433862.951596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433862.951600:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433862.951610:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433859/real 1789433862] req@ffff9811d2103b80 x1876350913769216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433862.951621:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913769216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433862.951628:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433862.951631:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433862.951634:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433862.951640:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433859/real 1789433862] req@ffff9811d2102a00 x1876350913769344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433862.951651:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913769344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433862.951657:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433862.951660:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433862.951663:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769344:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433862.951670:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433859/real 1789433862] req@ffff9811d2101500 x1876350913768960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433862.951681:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913768960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433862.951688:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433862.951692:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433862.951695:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913768960:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789433863.846086:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433864.870175:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433864.997997:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433864.998007:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433864.998014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433864.998032:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6234 00000100:00080000:1.0:1789433864.998040:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433864.998051:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433864.998079:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913769472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433864.998108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433864.998115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433864.998117:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433864.998121:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6234 00000100:00080000:1.0:1789433864.998124:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433864.998129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433864.998138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433864.998149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913769600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433864.998157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433864.998160:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433864.998162:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433864.998165:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6234 00000100:00080000:1.0:1789433864.998167:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433864.998171:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433864.998175:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433864.998184:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913769728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433864.998192:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433864.998195:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433864.998197:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433864.998200:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6234 00000100:00080000:1.0:1789433864.998203:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433864.998207:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433864.998211:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433864.998221:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913769856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433864.998353:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913769472:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433864.998406:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433864.998450:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433864/real 1789433864] req@ffff9811d0bdfb80 x1876350913769472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433864.998466:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913769472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433864.998476:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433864.998480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433864.998483:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913769472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433864.998490:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433864.998507:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433864.998518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433864/real 1789433864] req@ffff9811d0bdf480 x1876350913769600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433864.998528:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913769600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433864.998537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433864.998540:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433864.998542:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433864.998548:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433864.998561:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433864.998570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433864/real 1789433864] req@ffff9811d0bde680 x1876350913769728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433864.998581:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913769728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433864.998588:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433864.998591:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433864.998594:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433864.998599:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433864.998614:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433864.998624:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433864/real 1789433864] req@ffff9811d0bdd500 x1876350913769856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433864.998635:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913769856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433864.998642:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433864.998645:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433864.998648:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913769856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433865.894027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433866.918025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433867.942009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433868.966178:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433869.990009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433870.117986:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433870.117996:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433870.118000:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433870.118010:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6239 00000100:00080000:1.0:1789433870.118018:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433870.118030:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433870.118064:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913769984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433870.118102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433870.118106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433870.118108:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433870.118113:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6239 00000100:00080000:1.0:1789433870.118116:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433870.118122:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433870.118131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433870.118140:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913770112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433870.118147:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433870.118149:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433870.118150:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433870.118152:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6239 00000100:00080000:1.0:1789433870.118154:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433870.118157:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433870.118159:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433870.118164:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913770240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433870.118175:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433870.118177:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433870.118178:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433870.118180:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6239 00000100:00080000:1.0:1789433870.118181:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433870.118183:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433870.118185:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433870.118192:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913770368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433870.118270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913769984:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789433870.118323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433870.118362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433870/real 1789433870] req@ffff9811d0bdd180 x1876350913769984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433870.118385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913769984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433870.118411:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433870.118415:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433870.118418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913769984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433870.118426:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433870.118448:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433870.118461:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433870/real 1789433870] req@ffff9811d0bdc380 x1876350913770112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433870.118473:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913770112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433870.118480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433870.118483:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433870.118486:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433870.118492:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433870.118510:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433870.118521:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433870/real 1789433870] req@ffff9811d0bddc00 x1876350913770240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433870.118532:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913770240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433870.118558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433870.118561:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433870.118565:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433870.118571:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433870.118590:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433870.118601:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433870/real 1789433870] req@ffff9811d0bdd500 x1876350913770368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433870.118611:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913770368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433870.118617:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433870.118618:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433870.118620:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433871.014020:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433872.038291:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433873.062115:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433874.087588:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433875.110708:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433875.238040:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433875.238050:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433875.238054:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433875.238064:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6244 00000100:00080000:3.0:1789433875.238072:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433875.238084:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433875.238121:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913770496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433875.238148:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433875.238153:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433875.238155:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433875.238158:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6244 00000100:00080000:3.0:1789433875.238162:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433875.238167:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433875.238176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433875.238189:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913770624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433875.238199:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433875.238202:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433875.238204:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433875.238207:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6244 00000100:00080000:3.0:1789433875.238210:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433875.238214:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433875.238218:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433875.238229:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913770752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433875.238238:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433875.238241:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433875.238243:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433875.238247:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6244 00000100:00080000:3.0:1789433875.238250:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433875.238254:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433875.238259:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433875.238269:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913770880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433875.238333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913770496:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433875.238415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433875.238452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433875/real 1789433875] req@ffff9811d1bc7480 x1876350913770496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433875.238469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913770496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433875.238480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433875.238484:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433875.238487:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913770496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433875.238494:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433875.238515:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433875.238528:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433875/real 1789433875] req@ffff9811d1bc6a00 x1876350913770624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433875.238539:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913770624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433875.238547:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433875.238550:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433875.238553:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433875.238558:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433875.238576:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433875.238586:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433875/real 1789433875] req@ffff9811d1bc5180 x1876350913770752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433875.238597:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913770752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433875.238604:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433875.238607:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433875.238611:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433875.238617:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770880:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433875.238636:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433875.238653:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433875/real 1789433875] req@ffff9811d1bc7800 x1876350913770880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433875.238664:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913770880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433875.238672:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433875.238675:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433875.238678:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913770880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433876.134109:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433877.158081:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433878.182110:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433879.206168:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433880.229995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433880.357961:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433880.357984:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433880.357989:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433880.358001:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6249 00000100:00080000:3.0:1789433880.358013:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433880.358027:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433880.358076:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913771008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433880.358111:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433880.358115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433880.358117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433880.358122:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6249 00000100:00080000:3.0:1789433880.358126:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433880.358131:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433880.358142:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433880.358156:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913771136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433880.358167:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433880.358174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913771008:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433880.358180:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433880.358184:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433880.358188:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6249 00000100:00080000:3.0:1789433880.358192:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433880.358197:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433880.358203:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433880.358215:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913771264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433880.358226:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433880.358229:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433880.358231:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433880.358235:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6249 00000100:00080000:3.0:1789433880.358242:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433880.358253:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789433880.358257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789433880.358265:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433880.358280:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913771392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433880.358329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433880/real 1789433880] req@ffff9811d1bc4000 x1876350913771008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433880.358350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913771008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433880.358362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433880.358366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433880.358370:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913771008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433880.358397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433880.358420:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433880.358433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433880/real 1789433880] req@ffff9811d1bc5c00 x1876350913771136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433880.358451:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913771136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433880.358459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433880.358462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433880.358465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433880.358471:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433880.358491:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433880.358502:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433880/real 1789433880] req@ffff9811d1bc5880 x1876350913771264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433880.358513:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913771264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433880.358521:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433880.358524:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433880.358527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433880.358532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433880.358551:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433880.358562:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433880/real 1789433880] req@ffff9811d1bc5f80 x1876350913771392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433880.358572:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913771392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433880.358586:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433880.358589:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433880.358592:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433881.254052:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433882.278503:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433883.302019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433884.326302:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433885.350008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433885.477918:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433885.477928:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433885.477932:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433885.477941:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6255 00000100:00080000:3.0:1789433885.477950:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433885.477962:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433885.478007:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913771520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433885.478036:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433885.478040:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433885.478042:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433885.478046:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6255 00000100:00080000:3.0:1789433885.478050:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433885.478055:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433885.478065:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433885.478073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913771520:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433885.478080:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913771648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433885.478090:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433885.478094:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433885.478096:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433885.478100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6255 00000100:00080000:3.0:1789433885.478105:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433885.478110:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433885.478115:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433885.478126:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913771776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433885.478129:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433885.478141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433885.478145:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433885.478147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433885.478152:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6255 00000100:00080000:3.0:1789433885.478165:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433885.478173:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433885.478185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433885/real 1789433885] req@ffff9811d1bc6680 x1876350913771520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433885.478199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913771520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433885.478206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433885.478210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433885.478213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913771520:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433885.478235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433885.478257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433885.478269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433885/real 1789433885] req@ffff9811d1bc5500 x1876350913771648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433885.478296:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913771648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433885.478305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433885.478308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433885.478311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433885.478318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771776:192.168.203.128@tcp:38:kworker.0 00000008:00100000:3.0:1789433885.478330:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433885.478346:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913771904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433885.478403:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433885.478416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433885/real 1789433885] req@ffff9811d1bc6300 x1876350913771776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433885.478428:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913771776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433885.478435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433885.478438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433885.478441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433885.478455:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771904:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433885.478475:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433885.478497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433885/real 1789433885] req@ffff9811d1bc4700 x1876350913771904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433885.478508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913771904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433885.478516:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433885.478519:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433885.478522:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913771904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433886.374709:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433887.398097:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433888.422684:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433889.446492:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433890.470342:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433890.597933:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433890.597943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433890.597948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433890.597958:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6260 00000100:00080000:3.0:1789433890.597967:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433890.597978:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433890.598010:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913772032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433890.598041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433890.598045:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433890.598046:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433890.598050:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6260 00000100:00080000:3.0:1789433890.598054:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433890.598058:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433890.598068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433890.598082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913772160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433890.598097:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433890.598100:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433890.598101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433890.598104:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6260 00000100:00080000:3.0:1789433890.598108:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433890.598111:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433890.598116:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433890.598125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913772288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433890.598134:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433890.598137:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433890.598139:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433890.598142:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6260 00000100:00080000:3.0:1789433890.598145:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433890.598149:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433890.598152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433890.598161:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913772416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433890.598677:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913772032:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433890.598734:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433890.598793:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433890/real 1789433890] req@ffff9811d1bc7100 x1876350913772032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433890.598813:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913772032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433890.598824:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433890.598860:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433890.598866:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913772032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433890.598873:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433890.598897:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433890.598910:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433890/real 1789433890] req@ffff9811d1bc4700 x1876350913772160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433890.598922:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913772160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433890.598929:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433890.598932:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433890.598935:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433890.598941:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433890.598959:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433890.598970:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433890/real 1789433890] req@ffff9811d1bc6300 x1876350913772288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433890.598982:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913772288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433890.598996:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433890.598999:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433890.599002:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433890.599008:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433890.599027:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433890.599038:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433890/real 1789433890] req@ffff9811d1bc5500 x1876350913772416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433890.599049:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913772416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433890.599058:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433890.599061:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433890.599064:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433891.495567:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433892.518613:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433893.542396:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433894.566035:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433895.590017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433895.717910:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433895.717920:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433895.717926:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433895.717937:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6265 00000100:00080000:3.0:1789433895.717945:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433895.717956:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433895.718000:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913772544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433895.718025:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433895.718029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433895.718031:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433895.718036:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6265 00000100:00080000:3.0:1789433895.718039:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433895.718055:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433895.718065:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433895.718077:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913772672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433895.718086:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433895.718097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433895.718099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433895.718103:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6265 00000100:00080000:3.0:1789433895.718107:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433895.718111:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433895.718116:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433895.718127:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913772800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433895.718143:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433895.718145:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433895.718147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433895.718150:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6265 00000100:00080000:3.0:1789433895.718154:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433895.718158:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433895.718169:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433895.718181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913772928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433895.718197:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913772544:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433895.718268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433895.718310:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433895/real 1789433895] req@ffff9811d1bc6680 x1876350913772544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433895.718327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913772544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433895.718338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433895.718342:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433895.718346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913772544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433895.718353:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433895.718373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433895.718385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433895/real 1789433895] req@ffff9811d1bc5f80 x1876350913772672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433895.718397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913772672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433895.718404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433895.718407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433895.718411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433895.718416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433895.718433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433895.718451:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433895/real 1789433895] req@ffff9811d1bc5880 x1876350913772800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433895.718462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913772800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433895.718470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433895.718473:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433895.718476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433895.718497:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433895.718522:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433895.718532:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433895/real 1789433895] req@ffff9811d1bc5c00 x1876350913772928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433895.718542:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913772928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433895.718550:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433895.718552:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433895.718555:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913772928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433896.616264:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433897.638233:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433898.662112:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433899.688712:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433900.710145:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433900.837904:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433900.837913:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433900.837917:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433900.837927:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6270 00000100:00080000:3.0:1789433900.837934:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433900.837944:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433900.837976:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913773056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433900.838001:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433900.838004:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433900.838006:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433900.838013:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6270 00000100:00080000:3.0:1789433900.838017:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433900.838021:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433900.838030:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433900.838042:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913773184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433900.838053:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433900.838057:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913773056:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433900.838057:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433900.838061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433900.838068:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6270 00000100:00080000:3.0:1789433900.838071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433900.838074:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433900.838078:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433900.838089:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913773312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433900.838099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433900.838102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433900.838104:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433900.838107:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6270 00000100:00080000:3.0:1789433900.838112:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433900.838116:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433900.838121:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433900.838122:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433900.838147:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913773440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433900.838187:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433900/real 1789433900] req@ffff9811d1bc4000 x1876350913773056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433900.838206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913773056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433900.838216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433900.838220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433900.838224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913773056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433900.838244:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433900.838266:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433900.838279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433900/real 1789433900] req@ffff9811d1bc7800 x1876350913773184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433900.838290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913773184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433900.838304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433900.838307:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433900.838309:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433900.838315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433900.838333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433900.838343:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433900/real 1789433900] req@ffff9811d1bc5180 x1876350913773312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433900.838354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913773312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433900.838362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433900.838365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433900.838368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433900.838373:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773440:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433900.838391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433900.838400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433900/real 1789433900] req@ffff9811d1bc6a00 x1876350913773440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433900.838411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913773440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433900.838419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433900.838422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433900.838425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433901.734539:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433902.758088:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433903.782133:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433904.805960:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433905.830040:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433905.957900:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433905.957910:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433905.957914:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433905.957924:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6275 00000100:00080000:3.0:1789433905.957932:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433905.957942:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433905.957981:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913773568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433905.958007:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433905.958011:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433905.958012:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433905.958016:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6275 00000100:00080000:3.0:1789433905.958019:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433905.958023:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433905.958031:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433905.958043:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913773696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433905.958055:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789433905.958056:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913773568:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433905.958060:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433905.958069:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433905.958076:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6275 00000100:00080000:3.0:1789433905.958080:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433905.958085:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433905.958089:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433905.958102:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913773824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433905.958111:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433905.958117:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433905.958118:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433905.958122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433905.958136:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6275 00000100:00080000:3.0:1789433905.958141:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433905.958147:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433905.958152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433905.958165:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913773952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433905.958173:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433905/real 1789433905] req@ffff9811d1bc7480 x1876350913773568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433905.958194:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913773568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433905.958205:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433905.958210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433905.958213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913773568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433905.958235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433905.958256:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433905.958268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433905/real 1789433905] req@ffff9811d1bc6a00 x1876350913773696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433905.958276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913773696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433905.958281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433905.958283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433905.958285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433905.958289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433905.958300:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433905.958307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433905/real 1789433905] req@ffff9811d1bc5180 x1876350913773824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433905.958313:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913773824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433905.958322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433905.958326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433905.958329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433905.958332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433905.958343:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433905.958348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433905/real 1789433905] req@ffff9811d1bc7800 x1876350913773952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433905.958354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913773952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433905.958359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433905.958360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433905.958362:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913773952:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433906.854145:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433907.878022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433908.902007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433909.926045:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433910.950081:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433911.078032:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433911.078042:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433911.078046:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433911.078063:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6280 00000100:00080000:3.0:1789433911.078073:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433911.078084:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433911.078138:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913774080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433911.078168:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433911.078173:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433911.078175:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433911.078179:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6280 00000100:00080000:3.0:1789433911.078183:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433911.078188:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433911.078204:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433911.078219:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913774208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433911.078230:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433911.078233:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433911.078235:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433911.078239:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6280 00000100:00080000:3.0:1789433911.078243:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433911.078250:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433911.078260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913774080:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433911.078264:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433911.078287:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913774336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433911.078299:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433911.078303:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433911.078304:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433911.078309:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6280 00000100:00080000:3.0:1789433911.078314:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433911.078320:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433911.078327:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433911.078343:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913774464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433911.078350:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433911.078403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433911/real 1789433911] req@ffff9811d1bc4000 x1876350913774080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433911.078426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913774080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433911.078436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433911.078439:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433911.078442:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913774080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433911.078447:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433911.078464:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433911.078473:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433911/real 1789433911] req@ffff9811d1bc5c00 x1876350913774208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433911.078480:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913774208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433911.078485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433911.078487:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433911.078489:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433911.078509:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433911.078519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433911.078525:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433911/real 1789433911] req@ffff9811d1bc5880 x1876350913774336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433911.078531:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913774336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433911.078536:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433911.078537:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433911.078539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433911.078543:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774464:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433911.078553:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433911.078558:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433911/real 1789433911] req@ffff9811d1bc5f80 x1876350913774464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433911.078565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913774464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433911.078571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433911.078572:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433911.078574:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774464:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433911.973975:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433912.998023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433914.021959:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433915.046003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433916.070032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433916.197972:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433916.197985:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433916.197990:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433916.198008:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6285 00000100:00080000:3.0:1789433916.198019:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433916.198034:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433916.198096:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913774592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433916.198133:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433916.198137:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433916.198139:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433916.198144:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6285 00000100:00080000:3.0:1789433916.198148:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433916.198154:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433916.198165:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433916.198182:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913774720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433916.198202:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433916.198205:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433916.198209:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433916.198213:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6285 00000100:00080000:3.0:1789433916.198217:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433916.198222:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433916.198227:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433916.198240:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913774592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433916.198241:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913774848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433916.198258:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433916.198261:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433916.198263:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433916.198267:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6285 00000100:00080000:3.0:1789433916.198271:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433916.198275:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433916.198280:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433916.198303:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913774976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433916.198323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433916.198384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433916/real 1789433916] req@ffff9811d1bc6680 x1876350913774592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433916.198404:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913774592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433916.198416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433916.198420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433916.198424:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913774592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433916.198434:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433916.198460:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433916.198472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433916/real 1789433916] req@ffff9811d1bc5500 x1876350913774720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433916.198484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913774720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433916.198492:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433916.198495:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433916.198498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433916.198523:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433916.198543:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433916.198554:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433916/real 1789433916] req@ffff9811d1bc6300 x1876350913774848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433916.198566:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913774848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433916.198573:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433916.198576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433916.198580:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433916.198585:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774976:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433916.198604:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433916.198614:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433916/real 1789433916] req@ffff9811d1bc4700 x1876350913774976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433916.198625:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913774976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433916.198638:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433916.198641:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433916.198652:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913774976:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433917.094035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433918.118053:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433919.142112:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433920.166049:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433921.189991:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433921.317919:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433921.317927:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433921.317930:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433921.317938:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6290 00000100:00080000:3.0:1789433921.317945:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433921.317953:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433921.317980:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913775104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433921.318001:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433921.318003:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433921.318005:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433921.318007:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6290 00000100:00080000:3.0:1789433921.318021:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433921.318025:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433921.318033:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433921.318046:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913775232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433921.318054:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433921.318056:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433921.318057:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433921.318059:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6290 00000100:00080000:3.0:1789433921.318062:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433921.318064:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433921.318068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433921.318079:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913775104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433921.318081:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913775360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433921.318102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433921.318106:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433921.318108:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433921.318112:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6290 00000100:00080000:3.0:1789433921.318116:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433921.318122:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433921.318126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433921.318139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913775488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433921.318163:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433921.318212:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433921/real 1789433921] req@ffff9811d1bc7100 x1876350913775104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433921.318236:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913775104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433921.318249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433921.318252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433921.318255:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913775104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433921.318261:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433921.318276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433921.318284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433921/real 1789433921] req@ffff9811d1bc4700 x1876350913775232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433921.318291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913775232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433921.318297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433921.318299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433921.318301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433921.318322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433921.318333:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433921.318338:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433921/real 1789433921] req@ffff9811d1bc6300 x1876350913775360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433921.318345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913775360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433921.318353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433921.318356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433921.318359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433921.318365:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775488:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433921.318386:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433921.318397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433921/real 1789433921] req@ffff9811d1bc5500 x1876350913775488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433921.318418:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913775488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433921.318427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433921.318431:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433921.318434:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775488:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433922.214001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433923.238038:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433924.262017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433925.286003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433926.310176:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433926.437966:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433926.437985:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433926.437989:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433926.438001:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6296 00000100:00080000:3.0:1789433926.438011:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433926.438022:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433926.438053:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913775616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433926.438076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433926.438079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433926.438080:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433926.438083:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6296 00000100:00080000:3.0:1789433926.438086:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433926.438088:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433926.438096:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433926.438104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913775744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433926.438115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433926.438117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433926.438118:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433926.438123:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6296 00000100:00100000:0.0:1789433926.438124:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913775616:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433926.438125:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433926.438129:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433926.438133:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433926.438139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913775872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433926.438145:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433926.438154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433926.438155:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433926.438157:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6296 00000100:00080000:3.0:1789433926.438159:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433926.438162:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433926.438165:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433926.438174:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913776000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433926.438223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433926.438257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433926.438271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776000:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433927.334018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433928.358063:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433929.382025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789433929.510189:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789433929.510236:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789433929.510253:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433929.510294:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433929.510303:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789433929.510310:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789433929.510379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433926/real 1789433929] req@ffff9811d1bc5f80 x1876350913775744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433929.510404:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913775744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433929.510417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433929.510422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433929.510426:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433929.510433:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433926/real 1789433929] req@ffff9811d1bc5880 x1876350913775872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433929.510446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913775872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433929.510454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433929.510458:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433929.510461:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913775872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433929.510468:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433926/real 1789433929] req@ffff9811d1bc5c00 x1876350913776000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433929.510480:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913776000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433929.510489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433929.510492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433929.510495:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776000:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789433929.510501:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433926/real 1789433929] req@ffff9811d1bc6680 x1876350913775616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433929.510512:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913775616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433929.510519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433929.510522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433929.510525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913775616:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789433930.406009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433931.429981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433931.557943:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433931.557950:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433931.557953:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433931.557960:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6301 00000100:00080000:3.0:1789433931.557968:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433931.557977:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433931.558002:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913776128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433931.558023:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433931.558025:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433931.558026:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433931.558029:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6301 00000100:00080000:3.0:1789433931.558031:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433931.558034:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433931.558041:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433931.558049:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913776256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433931.558055:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433931.558060:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433931.558061:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433931.558063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6301 00000100:00080000:3.0:1789433931.558065:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433931.558069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433931.558070:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913776128:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433931.558073:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433931.558085:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913776384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433931.558096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433931.558099:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433931.558101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433931.558105:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6301 00000100:00080000:3.0:1789433931.558109:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433931.558113:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433931.558118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433931.558131:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433931.558132:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913776512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433931.558170:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433931/real 1789433931] req@ffff9811d1bc6680 x1876350913776128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433931.558187:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913776128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433931.558196:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433931.558200:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433931.558204:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913776128:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433931.558211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433931.558239:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433931.558252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433931/real 1789433931] req@ffff9811d1bc5c00 x1876350913776256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433931.558271:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913776256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433931.558280:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433931.558283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433931.558287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433931.558309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433931.558328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433931.558339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433931/real 1789433931] req@ffff9811d1bc5880 x1876350913776384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433931.558352:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913776384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433931.558360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433931.558363:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433931.558366:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433931.558372:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776512:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433931.558391:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433931.558401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433931/real 1789433931] req@ffff9811d1bc5f80 x1876350913776512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433931.558413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913776512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433931.558433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433931.558437:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433931.558441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776512:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433932.454017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433933.477980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433934.502026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433935.526010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433936.550033:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433936.677959:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433936.677969:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433936.677973:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433936.677983:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6306 00000100:00080000:3.0:1789433936.677991:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433936.678001:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433936.678037:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913776640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433936.678064:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433936.678067:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433936.678068:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433936.678072:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6306 00000100:00080000:3.0:1789433936.678074:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433936.678078:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433936.678085:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433936.678095:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913776768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433936.678102:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433936.678105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433936.678107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433936.678111:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6306 00000100:00080000:3.0:1789433936.678114:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433936.678119:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433936.678122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913776640:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789433936.678124:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433936.678139:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913776896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433936.678150:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433936.678154:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433936.678156:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433936.678160:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6306 00000100:00080000:3.0:1789433936.678164:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433936.678169:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433936.678174:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433936.678185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433936.678189:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913777024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433936.678234:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433936/real 1789433936] req@ffff9811d1bc4000 x1876350913776640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433936.678262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913776640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433936.678272:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433936.678276:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433936.678280:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913776640:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433936.678287:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433936.678308:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433936.678317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433936/real 1789433936] req@ffff9811d1bc7800 x1876350913776768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433936.678328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913776768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433936.678335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433936.678338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433936.678341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433936.678365:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433936.678384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433936.678395:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433936/real 1789433936] req@ffff9811d1bc5180 x1876350913776896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433936.678406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913776896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433936.678413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433936.678417:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433936.678420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913776896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433936.678425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777024:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433936.678449:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433936.678465:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433936/real 1789433936] req@ffff9811d1bc6a00 x1876350913777024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433936.678477:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913777024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433936.678485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433936.678488:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433936.678491:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777024:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433937.574015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433938.598017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433939.622073:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433940.646094:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433941.670007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789433941.798000:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433941.798010:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789433941.798014:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433941.798024:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6311 00000100:00080000:2.0:1789433941.798034:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433941.798045:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789433941.798077:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913777152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433941.798102:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433941.798106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789433941.798107:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433941.798112:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6311 00000100:00080000:2.0:1789433941.798116:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433941.798129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433941.798137:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913777152:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789433941.798140:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433941.798158:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913777280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789433941.798169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433941.798173:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789433941.798176:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433941.798179:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6311 00000100:00080000:2.0:1789433941.798183:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433941.798189:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433941.798194:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789433941.798207:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913777408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789433941.798211:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789433941.798222:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789433941.798225:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789433941.798226:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789433941.798231:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6311 00000100:00080000:2.0:1789433941.798235:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789433941.798248:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789433941.798253:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433941.798259:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433941/real 1789433941] req@ffff9811d2102d80 x1876350913777152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789433941.798271:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913777536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433941.798282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913777152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433941.798293:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433941.798299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433941.798303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913777152:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433941.798325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433941.798348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433941.798361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433941/real 1789433941] req@ffff9811d2103800 x1876350913777280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433941.798373:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913777280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433941.798380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433941.798383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433941.798386:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433941.798407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433941.798425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433941.798436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433941/real 1789433941] req@ffff9811d2100000 x1876350913777408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433941.798446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913777408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433941.798459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433941.798461:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433941.798465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433941.798471:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777536:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433941.798490:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433941.798500:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433941/real 1789433941] req@ffff9811d2101500 x1876350913777536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433941.798510:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913777536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433941.798518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433941.798520:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433941.798523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777536:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433942.694025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433943.718008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433944.742000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433945.765980:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433946.790464:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433946.918063:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433946.918072:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433946.918076:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433946.918094:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6316 00000100:00080000:3.0:1789433946.918102:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433946.918113:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433946.918148:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913777664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433946.918180:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433946.918184:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433946.918185:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433946.918190:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6316 00000100:00080000:3.0:1789433946.918193:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433946.918197:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433946.918207:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433946.918230:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913777792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433946.918239:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433946.918242:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433946.918244:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433946.918247:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6316 00000100:00080000:3.0:1789433946.918250:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433946.918253:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433946.918258:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433946.918276:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913777920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433946.918286:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433946.918289:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433946.918290:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433946.918293:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6316 00000100:00080000:3.0:1789433946.918296:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433946.918300:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433946.918304:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433946.918315:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913778048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433946.918394:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913777664:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433946.918460:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433946.918508:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433946/real 1789433946] req@ffff9811d1bc7480 x1876350913777664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433946.918526:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913777664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433946.918536:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433946.918540:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433946.918544:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913777664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433946.918566:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433946.918591:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433946.918615:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433946/real 1789433946] req@ffff9811d1bc6a00 x1876350913777792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433946.918627:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913777792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433946.918636:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433946.918639:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433946.918643:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433946.918649:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433946.918679:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433946.918691:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433946/real 1789433946] req@ffff9811d1bc5180 x1876350913777920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433946.918702:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913777920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433946.918710:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433946.918713:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433946.918717:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913777920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433946.918723:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778048:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433946.918742:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433946.918753:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433946/real 1789433946] req@ffff9811d1bc7800 x1876350913778048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433946.918765:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913778048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433946.918788:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433946.918792:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433946.918795:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778048:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433947.814258:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433948.837967:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433949.862022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433950.886002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433951.910042:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433952.037948:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433952.037957:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433952.037960:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433952.037967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6321 00000100:00080000:1.0:1789433952.037974:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433952.037983:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433952.038016:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913778176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433952.038040:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433952.038043:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433952.038044:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433952.038052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6321 00000100:00080000:1.0:1789433952.038055:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433952.038059:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433952.038069:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433952.038095:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913778176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433952.038095:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913778304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433952.038111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433952.038114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433952.038116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433952.038120:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6321 00000100:00080000:1.0:1789433952.038123:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433952.038128:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433952.038134:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433952.038144:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913778432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433952.038153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433952.038156:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1789433952.038159:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433952.038160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433952.038166:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6321 00000100:00080000:1.0:1789433952.038171:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433952.038177:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433952.038190:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433952.038204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433952/real 1789433952] req@ffff9811d0bde680 x1876350913778176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789433952.038207:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913778560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433952.038238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913778176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433952.038250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433952.038253:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433952.038257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913778176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433952.038277:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433952.038299:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433952.038308:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433952/real 1789433952] req@ffff9811d0bdf480 x1876350913778304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433952.038316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913778304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433952.038325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433952.038327:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433952.038329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433952.038335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433952.038352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433952.038362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433952/real 1789433952] req@ffff9811d0bdfb80 x1876350913778432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433952.038374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913778432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433952.038381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433952.038384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433952.038388:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433952.038392:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778560:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433952.038407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433952.038413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433952/real 1789433952] req@ffff9811d0bded80 x1876350913778560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433952.038420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913778560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433952.038425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433952.038427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433952.038429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433952.934000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433953.958038:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433954.982018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433956.006652:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433957.030028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433957.158028:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433957.158039:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433957.158042:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433957.158052:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6326 00000100:00080000:3.0:1789433957.158060:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433957.158071:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433957.158107:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913778688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433957.158137:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433957.158141:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433957.158143:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433957.158147:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6326 00000100:00080000:3.0:1789433957.158150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433957.158155:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433957.158165:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433957.158178:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913778816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433957.158188:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433957.158191:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433957.158192:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433957.158195:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6326 00000100:00080000:3.0:1789433957.158199:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433957.158203:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433957.158207:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433957.158217:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913778944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433957.158226:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433957.158230:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433957.158231:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433957.158235:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6326 00000100:00080000:3.0:1789433957.158237:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433957.158242:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433957.158246:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433957.158257:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913779072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433957.158399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913778688:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433957.158452:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433957.158486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433957/real 1789433957] req@ffff9811d1bc4000 x1876350913778688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433957.158504:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913778688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433957.158514:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433957.158517:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433957.158521:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913778688:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433957.158527:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433957.158548:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433957.158559:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433957/real 1789433957] req@ffff9811d1bc5f80 x1876350913778816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433957.158571:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913778816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433957.158579:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433957.158582:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433957.158585:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433957.158590:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433957.158608:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433957.158618:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433957/real 1789433957] req@ffff9811d1bc5880 x1876350913778944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433957.158629:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913778944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433957.158636:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433957.158639:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433957.158642:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913778944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433957.158648:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779072:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433957.158665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433957.158675:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433957/real 1789433957] req@ffff9811d1bc5c00 x1876350913779072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433957.158685:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913779072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433957.158693:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433957.158696:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433957.158698:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779072:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433958.054031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433959.078001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433960.102428:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433961.126012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433962.150452:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433962.277924:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433962.277932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433962.277936:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433962.277946:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6331 00000100:00080000:3.0:1789433962.277954:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433962.277966:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433962.278001:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913779200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433962.278025:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433962.278028:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433962.278029:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433962.278033:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6331 00000100:00080000:3.0:1789433962.278036:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433962.278041:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433962.278049:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433962.278059:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913779328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433962.278077:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913779200:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789433962.278079:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433962.278083:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433962.278086:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433962.278092:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6331 00000100:00080000:3.0:1789433962.278096:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433962.278100:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433962.278105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433962.278117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913779456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433962.278125:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433962.278128:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433962.278130:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433962.278134:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6331 00000100:00080000:3.0:1789433962.278138:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789433962.278143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789433962.278144:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433962.278152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433962.278164:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913779584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433962.278189:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433962/real 1789433962] req@ffff9811d1bc6680 x1876350913779200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433962.278219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913779200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433962.278230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433962.278234:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433962.278237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913779200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433962.278260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433962.278282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433962.278293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433962/real 1789433962] req@ffff9811d1bc5500 x1876350913779328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433962.278305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913779328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433962.278328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433962.278331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433962.278335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433962.278340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433962.278359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433962.278368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433962/real 1789433962] req@ffff9811d1bc6300 x1876350913779456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433962.278378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913779456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433962.278384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433962.278387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433962.278389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433962.278395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779584:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433962.278413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433962.278423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433962/real 1789433962] req@ffff9811d1bc4700 x1876350913779584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433962.278434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913779584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433962.278442:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433962.278445:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433962.278448:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779584:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433963.175095:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433964.198103:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433965.222019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433966.246006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433967.270105:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433967.399862:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433967.399873:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433967.399878:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433967.399890:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6336 00000100:00080000:1.0:1789433967.399903:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433967.399914:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433967.399956:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913779712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433967.399981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433967.399984:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433967.399985:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433967.399988:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6336 00000100:00080000:1.0:1789433967.399990:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433967.399993:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433967.400003:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433967.400011:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913779840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433967.400017:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433967.400020:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433967.400021:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433967.400023:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6336 00000100:00080000:1.0:1789433967.400024:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433967.400027:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433967.400029:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433967.400034:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913779968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433967.400045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433967.400048:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433967.400049:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433967.400051:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6336 00000100:00080000:1.0:1789433967.400052:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433967.400054:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433967.400057:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433967.400062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913780096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433967.400382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913779712:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433967.400444:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433967.400531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433967/real 1789433967] req@ffff9811d0bdc000 x1876350913779712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433967.400546:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913779712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433967.400553:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433967.400556:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433967.400558:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913779712:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433967.400564:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433967.400580:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433967.400588:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433967/real 1789433967] req@ffff9811d0bdc700 x1876350913779840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433967.400595:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913779840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433967.400600:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433967.400602:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433967.400604:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433967.400607:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433967.400617:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433967.400621:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433967/real 1789433967] req@ffff9811d0bdea00 x1876350913779968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433967.400628:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913779968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433967.400632:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433967.400634:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433967.400635:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913779968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433967.400639:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780096:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433967.400666:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433967.400687:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433967/real 1789433967] req@ffff9811d0bdd880 x1876350913780096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433967.400698:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913780096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433967.400711:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433967.400714:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433967.400716:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780096:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433968.294007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433969.318033:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433970.341969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433971.366001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433972.389938:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433972.517945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433972.517954:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433972.517958:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433972.517967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6342 00000100:00080000:1.0:1789433972.517975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433972.517986:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433972.518018:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913780224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433972.518041:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433972.518045:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433972.518047:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433972.518051:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6342 00000100:00080000:1.0:1789433972.518054:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433972.518059:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433972.518069:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433972.518079:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913780352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433972.518085:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913780224:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433972.518088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433972.518093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433972.518094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433972.518097:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6342 00000100:00080000:1.0:1789433972.518099:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433972.518102:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433972.518105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433972.518112:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913780480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433972.518119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433972.518121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433972.518123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433972.518125:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6342 00000100:00080000:1.0:1789433972.518143:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433972.518148:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433972.518153:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789433972.518158:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433972.518166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913780608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433972.518201:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433972/real 1789433972] req@ffff9811d0bdf800 x1876350913780224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433972.518219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913780224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433972.518230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433972.518234:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433972.518238:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913780224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433972.518257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433972.518274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433972.518282:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433972/real 1789433972] req@ffff9811d0bde300 x1876350913780352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433972.518290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913780352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433972.518295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433972.518297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433972.518299:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433972.518303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433972.518312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433972.518317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433972/real 1789433972] req@ffff9811d0bdf100 x1876350913780480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433972.518324:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913780480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433972.518329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433972.518331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433972.518334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433972.518338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780608:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433972.518355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433972.518364:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433972/real 1789433972] req@ffff9811d0bdd880 x1876350913780608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433972.518376:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913780608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789433997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433972.518382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433972.518384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433972.518386:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780608:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433973.413996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433974.438011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433975.461995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433976.486042:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433977.510047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433977.639147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433977.639157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433977.639161:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433977.639171:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6347 00000100:00080000:1.0:1789433977.639180:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433977.639193:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433977.639227:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913780736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433977.639255:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433977.639258:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433977.639260:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433977.639264:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6347 00000100:00080000:1.0:1789433977.639268:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433977.639273:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433977.639283:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433977.639296:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913780864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433977.639307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913780736:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789433977.639310:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433977.639316:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433977.639318:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433977.639322:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6347 00000100:00080000:1.0:1789433977.639332:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433977.639337:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433977.639342:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433977.639354:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913780992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433977.639371:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789433977.639372:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789433977.639376:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433977.639378:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433977.639382:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6347 00000100:00080000:1.0:1789433977.639386:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433977.639390:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433977.639394:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433977.639405:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913781120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433977.639647:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433977/real 1789433977] req@ffff9811d0bdea00 x1876350913780736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433977.639665:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913780736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433977.639675:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433977.639678:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433977.639682:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913780736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433977.639704:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433977.639727:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433977.639740:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433977/real 1789433977] req@ffff9811d0bdc700 x1876350913780864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433977.639752:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913780864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433977.639760:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433977.639764:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433977.639767:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433977.639772:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433977.639804:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433977.639816:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433977/real 1789433977] req@ffff9811d0bdc000 x1876350913780992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433977.639828:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913780992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433977.639878:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433977.639882:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433977.639885:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913780992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433977.639891:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781120:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433977.639912:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433977.639923:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433977/real 1789433977] req@ffff9811d0bded80 x1876350913781120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433977.639934:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913781120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433977.639942:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433977.639945:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433977.639948:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781120:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433978.534072:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433979.558010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433980.582019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433981.607581:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433982.630000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789433982.757941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433982.757949:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789433982.757953:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433982.757961:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6352 00000100:00080000:0.0:1789433982.757973:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433982.757982:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789433982.758011:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913781248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433982.758036:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433982.758039:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789433982.758040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433982.758043:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6352 00000100:00080000:0.0:1789433982.758044:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433982.758047:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433982.758054:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433982.758062:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913781376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789433982.758068:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433982.758070:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789433982.758071:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433982.758073:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6352 00000100:00080000:0.0:1789433982.758075:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433982.758077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433982.758080:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789433982.758087:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913781504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433982.758090:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913781248:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789433982.758097:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789433982.758100:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789433982.758101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789433982.758103:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6352 00000100:00080000:0.0:1789433982.758105:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789433982.758107:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789433982.758110:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789433982.758117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913781632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789433982.758165:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433982.758199:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433982/real 1789433982] req@ffff9811c8cd4000 x1876350913781248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433982.758217:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913781248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433982.758227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433982.758231:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433982.758235:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913781248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433982.758242:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433982.758263:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433982.758275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433982/real 1789433982] req@ffff9811c8cd6300 x1876350913781376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433982.758298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913781376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433982.758306:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433982.758309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433982.758313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433982.758334:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433982.758355:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433982.758367:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433982/real 1789433982] req@ffff9811c8cd7b80 x1876350913781504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433982.758379:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913781504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433982.758387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433982.758390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433982.758393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433982.758399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781632:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433982.758432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433982.758443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433982/real 1789433982] req@ffff9811c8cd4a80 x1876350913781632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433982.758455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913781632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433982.758463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433982.758466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433982.758469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781632:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789433983.653977:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433984.677991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433985.701987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433986.726045:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433987.749943:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433987.877938:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433987.877947:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433987.877951:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433987.877962:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6357 00000100:00080000:1.0:1789433987.877970:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433987.877982:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433987.878014:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913781760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433987.878041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433987.878045:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433987.878047:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433987.878051:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6357 00000100:00080000:1.0:1789433987.878054:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433987.878059:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433987.878069:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433987.878078:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913781760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433987.878084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913781888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433987.878096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433987.878099:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433987.878101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433987.878105:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6357 00000100:00080000:1.0:1789433987.878109:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433987.878113:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433987.878118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789433987.878130:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789433987.878131:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913782016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433987.878143:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433987.878147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433987.878149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433987.878152:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6357 00000100:00080000:1.0:1789433987.878156:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433987.878163:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433987.878163:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433987/real 1789433987] req@ffff9811d0bdfb80 x1876350913781760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789433987.878171:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433987.878185:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913782144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433987.878188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913781760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433987.878212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433987.878216:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433987.878221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913781760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789433987.878243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433987.878265:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433987.878277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433987/real 1789433987] req@ffff9811d0bdf480 x1876350913781888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433987.878290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913781888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433987.878297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433987.878301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433987.878304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913781888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433987.878309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789433987.878328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433987.878340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433987/real 1789433987] req@ffff9811d0bde680 x1876350913782016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433987.878357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913782016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433987.878364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433987.878368:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433987.878371:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433987.878376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782144:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789433987.878394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789433987.878404:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433987/real 1789433987] req@ffff9811d0bdd500 x1876350913782144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789433987.878416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913782144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789433987.878423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789433987.878426:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789433987.878429:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782144:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433988.774005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433989.797962:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789433990.821994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433991.845942:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433992.870004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789433992.997908:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433992.997915:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789433992.997919:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433992.997928:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6362 00000100:00080000:1.0:1789433992.997935:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433992.997943:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789433992.997970:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913782272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433992.997987:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433992.997990:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789433992.997991:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433992.997993:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6362 00000100:00080000:1.0:1789433992.997995:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433992.997997:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433992.998004:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789433992.998013:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913782400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433992.998018:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433992.998021:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789433992.998022:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433992.998024:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6362 00000100:00080000:1.0:1789433992.998025:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433992.998029:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433992.998033:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433992.998033:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913782272:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789433992.998041:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913782528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789433992.998052:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789433992.998054:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789433992.998055:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789433992.998057:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6362 00000100:00080000:1.0:1789433992.998059:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789433992.998062:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789433992.998064:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789433992.998072:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913782656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789433992.998122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433992.998164:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789433992.998186:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782656:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789433993.893939:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433994.918052:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789433995.941969:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789433996.070141:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789433996.070185:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789433996.070201:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433996.070239:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433996.070248:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789433996.070259:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789433996.070270:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433992/real 1789433996] req@ffff9811d0bdd180 x1876350913782528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433996.070296:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913782528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433996.070308:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433996.070313:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433996.070317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433996.070335:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433992/real 1789433996] req@ffff9811d0bdd500 x1876350913782656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433996.070348:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913782656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433996.070356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433996.070359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433996.070363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782656:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789433996.070369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433992/real 1789433996] req@ffff9811d0bddc00 x1876350913782272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433996.070380:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913782272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433996.070387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433996.070390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433996.070393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913782272:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433996.070399:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433992/real 1789433996] req@ffff9811d0bdc380 x1876350913782400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433996.070410:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913782400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433996.070417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433996.070420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433996.070423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782400:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789433996.966007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789433997.989987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789433998.118162:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433998.118172:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789433998.118176:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433998.118187:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6367 00000100:00080000:3.0:1789433998.118195:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433998.118213:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789433998.118249:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913782784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433998.118279:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433998.118283:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789433998.118284:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433998.118288:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6367 00000100:00080000:3.0:1789433998.118291:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433998.118295:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433998.118305:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433998.118319:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913782912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433998.118332:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433998.118335:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789433998.118336:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433998.118339:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6367 00000100:00080000:3.0:1789433998.118342:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433998.118346:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433998.118350:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789433998.118359:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913783040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789433998.118368:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789433998.118371:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789433998.118372:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789433998.118375:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6367 00000100:00080000:3.0:1789433998.118378:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789433998.118381:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789433998.118385:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789433998.118395:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913783168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789433998.118901:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913782784:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789433998.118964:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433998.119022:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433998/real 1789433998] req@ffff9811d1bc7100 x1876350913782784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433998.119039:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913782784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433998.119055:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433998.119059:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433998.119063:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913782784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789433998.119071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433998.119097:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433998.119108:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433998/real 1789433998] req@ffff9811d1bc4700 x1876350913782912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433998.119119:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913782912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433998.119127:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433998.119130:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433998.119133:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913782912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433998.119138:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789433998.119157:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433998.119167:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433998/real 1789433998] req@ffff9811d1bc6300 x1876350913783040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433998.119177:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913783040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433998.119184:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433998.119186:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433998.119189:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789433998.119196:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783168:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789433998.119214:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789433998.119224:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789433998/real 1789433998] req@ffff9811d1bc5500 x1876350913783168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789433998.119234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913783168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789433998.119241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789433998.119243:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789433998.119246:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783168:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789433999.013998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434000.038000:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434001.061986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434002.085999:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434003.109998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434003.237959:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434003.237970:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434003.237974:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434003.237984:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6372 00000100:00080000:3.0:1789434003.237993:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434003.238004:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434003.238037:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913783296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434003.238061:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434003.238064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434003.238065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434003.238070:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6372 00000100:00080000:3.0:1789434003.238073:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434003.238077:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434003.238087:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913783296:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789434003.238087:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434003.238102:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913783424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434003.238112:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434003.238115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434003.238116:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434003.238120:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6372 00000100:00080000:3.0:1789434003.238124:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434003.238137:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434003.238141:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789434003.238145:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434003.238159:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913783552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434003.238168:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434003.238172:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434003.238175:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434003.238176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434003/real 1789434003] req@ffff9811d1bc6680 x1876350913783296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434003.238182:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6372 00000100:00080000:3.0:1789434003.238190:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789434003.238196:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913783296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434003.238197:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434003.238206:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789434003.238208:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434003.238212:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434003.238215:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913783296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434003.238221:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913783680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434003.238235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434003.238262:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434003.238269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434003/real 1789434003] req@ffff9811d1bc5c00 x1876350913783424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434003.238279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913783424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434003.238287:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434003.238290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434003.238293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434003.238299:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434003.238316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434003.238326:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434003/real 1789434003] req@ffff9811d1bc5880 x1876350913783552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434003.238343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913783552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434003.238350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434003.238352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434003.238355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434003.238359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783680:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434003.238372:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434003.238378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434003/real 1789434003] req@ffff9811d1bc5f80 x1876350913783680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434003.238389:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913783680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434003.238394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434003.238395:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434003.238397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783680:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434004.133993:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434005.157984:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434006.181970:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434007.205968:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434008.229992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434008.357945:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434008.357954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434008.357958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434008.357967:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6377 00000100:00080000:3.0:1789434008.357975:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434008.357987:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434008.358021:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913783808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434008.358040:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434008.358042:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434008.358044:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434008.358047:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6377 00000100:00080000:3.0:1789434008.358051:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434008.358057:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434008.358067:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434008.358071:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913783808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434008.358083:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913783936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434008.358095:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434008.358098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434008.358100:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434008.358104:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6377 00000100:00080000:3.0:1789434008.358108:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434008.358113:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434008.358118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789434008.358127:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434008.358132:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913784064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434008.358144:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434008.358147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434008.358149:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434008.358153:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6377 00000100:00080000:3.0:1789434008.358157:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434008.358162:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434008.358166:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434008/real 1789434008] req@ffff9811d1bc4000 x1876350913783808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789434008.358169:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434008.358188:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913784192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434008.358190:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913783808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434008.358219:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434008.358224:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434008.358229:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913783808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434008.358250:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434008.358269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434008.358280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434008/real 1789434008] req@ffff9811d1bc7800 x1876350913783936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434008.358291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913783936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434008.358305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434008.358308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434008.358311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913783936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434008.358316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434008.358344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434008.358353:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434008/real 1789434008] req@ffff9811d1bc5180 x1876350913784064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434008.358364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913784064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434008.358387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434008.358391:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434008.358400:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434008.358406:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784192:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434008.358432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434008.358442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434008/real 1789434008] req@ffff9811d1bc6a00 x1876350913784192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434008.358453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913784192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434008.358460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434008.358463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434008.358466:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434009.254126:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434010.278012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434011.301947:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434012.325999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434013.349981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434013.477941:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434013.477952:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434013.477956:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434013.477966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6383 00000100:00080000:2.0:1789434013.477974:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434013.477985:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434013.478019:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913784320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434013.478046:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434013.478049:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434013.478051:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434013.478055:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6383 00000100:00080000:2.0:1789434013.478059:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434013.478064:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434013.478075:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434013.478078:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913784320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434013.478090:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913784448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434013.478101:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434013.478105:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434013.478107:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434013.478111:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6383 00000100:00080000:2.0:1789434013.478115:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434013.478121:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434013.478126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434013.478136:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913784576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434013.478142:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434013.478161:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434013.478164:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434013.478166:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434013.478170:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6383 00000100:00080000:2.0:1789434013.478174:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789434013.478179:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434013/real 1789434013] req@ffff9811d2102a00 x1876350913784320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789434013.478183:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434013.478191:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434013.478204:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913784320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789434013.478207:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913784704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434013.478228:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434013.478233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434013.478236:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913784320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434013.478255:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434013.478271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434013.478280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434013/real 1789434013] req@ffff9811d2103b80 x1876350913784448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434013.478290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913784448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434013.478297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434013.478300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434013.478303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434013.478309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434013.478347:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434013.478359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434013/real 1789434013] req@ffff9811d2101c00 x1876350913784576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434013.478371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913784576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434013.478376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434013.478379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434013.478380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434013.478384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784704:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434013.478398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434013.478404:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434013/real 1789434013] req@ffff9811d2101180 x1876350913784704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434013.478411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913784704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434013.478416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434013.478418:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434013.478420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784704:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434014.374006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434015.398856:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434016.422008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434017.446032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434018.469998:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434018.597977:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434018.597987:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434018.597991:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434018.598001:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6388 00000100:00080000:2.0:1789434018.598009:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434018.598020:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434018.598056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913784832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434018.598096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434018.598100:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434018.598101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434018.598106:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6388 00000100:00080000:2.0:1789434018.598110:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434018.598128:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434018.598131:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913784832:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434018.598189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434018.598239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434018/real 1789434018] req@ffff9811d2101880 x1876350913784832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434018.598259:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913784832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434018.598269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434018.598274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434018.598277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913784832:192.168.203.128@tcp:250:kworker.0 00000008:00100000:2.0:1789434018.598311:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434018.598336:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913784960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434018.598355:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434018.598358:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434018.598360:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434018.598364:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6388 00000100:00080000:2.0:1789434018.598368:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434018.598373:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434018.598379:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434018.598388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789434018.598393:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913785088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434018.598406:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434018.598409:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434018.598412:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789434018.598413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434018.598422:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6388 00000100:00100000:0.0:1789434018.598427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434018/real 1789434018] req@ffff9811d2100e00 x1876350913784960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789434018.598429:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434018.598440:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434018.598446:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434018.598452:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913784960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789434018.598461:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913785216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434018.598463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434018.598471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434018.598478:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913784960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434018.598503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434018.598525:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434018.598546:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434018/real 1789434018] req@ffff9811d2101f80 x1876350913785088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434018.598558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913785088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434018.598573:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434018.598576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434018.598587:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434018.598594:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785216:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434018.598615:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434018.598624:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434018/real 1789434018] req@ffff9811d2101880 x1876350913785216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434018.598631:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913785216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434018.598637:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434018.598650:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434018.598654:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434019.494067:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434020.518037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434021.542013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434022.566002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434023.589979:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434023.717981:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434023.717989:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434023.717992:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434023.718002:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6393 00000100:00080000:1.0:1789434023.718010:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434023.718020:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434023.718052:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913785344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434023.718076:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434023.718079:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434023.718081:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434023.718085:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6393 00000100:00080000:1.0:1789434023.718088:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434023.718094:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434023.718104:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434023.718115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913785344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434023.718118:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913785472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434023.718131:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434023.718134:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434023.718136:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434023.718138:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6393 00000100:00080000:1.0:1789434023.718141:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434023.718145:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434023.718150:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434023.718162:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913785600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434023.718172:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434023.718176:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434023.718178:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434023.718181:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6393 00000100:00080000:1.0:1789434023.718185:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789434023.718190:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434023.718193:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434023.718198:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434023.718208:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913785728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434023.718239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434023/real 1789434023] req@ffff9811d0bde680 x1876350913785344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434023.718268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913785344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434023.718279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434023.718283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434023.718287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913785344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434023.718308:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434023.718337:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434023.718350:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434023/real 1789434023] req@ffff9811d0bdf480 x1876350913785472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434023.718361:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913785472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434023.718368:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434023.718372:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434023.718375:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434023.718380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434023.718398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434023.718423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434023/real 1789434023] req@ffff9811d0bdfb80 x1876350913785600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434023.718438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913785600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434023.718445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434023.718448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434023.718451:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434023.718456:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785728:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434023.718475:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434023.718486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434023/real 1789434023] req@ffff9811d0bded80 x1876350913785728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434023.718496:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913785728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434023.718511:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434023.718514:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434023.718517:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785728:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434024.614022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434025.638298:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434026.661970:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434027.686006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434028.709962:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434028.837976:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434028.837988:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434028.837992:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434028.838003:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6398 00000100:00080000:3.0:1789434028.838012:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434028.838024:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434028.838060:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913785856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434028.838091:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434028.838095:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434028.838096:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434028.838100:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6398 00000100:00080000:3.0:1789434028.838105:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434028.838109:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434028.838119:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434028.838132:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913785984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434028.838141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434028.838144:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434028.838146:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434028.838148:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6398 00000100:00080000:3.0:1789434028.838151:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434028.838155:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434028.838159:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434028.838168:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913786112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434028.838177:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434028.838181:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434028.838182:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434028.838185:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6398 00000100:00080000:3.0:1789434028.838188:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434028.838192:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434028.838196:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434028.838205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913786240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434028.838285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913785856:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434028.838346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434028.838384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434028/real 1789434028] req@ffff9811d1bc7480 x1876350913785856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434028.838416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913785856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434028.838426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434028.838429:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434028.838433:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913785856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434028.838439:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434028.838463:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434028.838474:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434028/real 1789434028] req@ffff9811d1bc6a00 x1876350913785984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434028.838491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913785984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434028.838499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434028.838502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434028.838505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913785984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434028.838510:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434028.838527:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434028.838537:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434028/real 1789434028] req@ffff9811d1bc5180 x1876350913786112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434028.838548:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913786112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434028.838555:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434028.838558:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434028.838560:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434028.838566:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786240:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434028.838586:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434028.838595:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434028/real 1789434028] req@ffff9811d1bc7800 x1876350913786240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434028.838604:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913786240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434028.838617:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434028.838620:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434028.838623:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786240:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434029.734115:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434030.757976:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434031.782101:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434032.806415:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434033.830159:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434033.960289:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434033.960299:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434033.960303:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434033.960313:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6403 00000100:00080000:3.0:1789434033.960323:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434033.960333:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434033.960368:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913786368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434033.960404:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434033.960407:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434033.960409:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434033.960414:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6403 00000100:00080000:3.0:1789434033.960417:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434033.960422:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434033.960432:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434033.960445:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913786496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434033.960456:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434033.960459:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434033.960461:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434033.960464:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6403 00000100:00080000:3.0:1789434033.960468:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434033.960472:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434033.960476:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434033.960487:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913786624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434033.960496:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434033.960500:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434033.960501:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434033.960505:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6403 00000100:00080000:3.0:1789434033.960508:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434033.960512:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434033.960517:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434033.960529:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913786752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434033.960625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913786368:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434033.960683:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434033.960726:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434033/real 1789434033] req@ffff9811d1bc4000 x1876350913786368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434033.960744:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913786368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434033.960754:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434033.960758:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434033.960762:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913786368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434033.960768:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434033.960790:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434033.960802:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434033/real 1789434033] req@ffff9811d1bc5f80 x1876350913786496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434033.960814:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913786496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434033.960822:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434033.960825:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434033.960829:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434033.960859:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434033.960880:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434033.960892:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434033/real 1789434033] req@ffff9811d1bc5880 x1876350913786624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434033.960903:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913786624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434033.960912:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434033.960915:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434033.960917:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434033.960924:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786752:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434033.960942:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434033.960952:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434033/real 1789434033] req@ffff9811d1bc5c00 x1876350913786752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434033.960963:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913786752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434033.960971:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434033.960975:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434033.960978:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913786752:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434034.853997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434035.878028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434036.902930:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434037.926071:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434038.950023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434039.077920:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434039.077930:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434039.077934:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434039.077954:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6408 00000100:00080000:3.0:1789434039.077964:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434039.077976:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434039.078009:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913786880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434039.078041:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434039.078044:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434039.078058:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434039.078063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6408 00000100:00080000:3.0:1789434039.078066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434039.078072:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434039.078081:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434039.078097:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913787008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434039.078098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913786880:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434039.078114:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434039.078118:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434039.078120:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434039.078124:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6408 00000100:00080000:3.0:1789434039.078128:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434039.078132:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434039.078154:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434039.078168:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913787136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434039.078177:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434039.078185:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434039.078194:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434039.078195:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434039.078200:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6408 00000100:00080000:3.0:1789434039.078205:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434039.078213:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434039.078214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434039/real 1789434039] req@ffff9811d1bc6680 x1876350913786880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789434039.078227:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434039.078234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913786880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434039.078244:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913787264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434039.078247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434039.078255:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434039.078263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913786880:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434039.078284:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434039.078302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434039.078312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434039/real 1789434039] req@ffff9811d1bc5500 x1876350913787008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434039.078323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913787008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434039.078330:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434039.078333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434039.078336:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434039.078341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434039.078354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434039.078362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434039/real 1789434039] req@ffff9811d1bc6300 x1876350913787136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434039.078379:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913787136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434039.078386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434039.078388:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434039.078391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434039.078396:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787264:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434039.078411:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434039.078420:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434039/real 1789434039] req@ffff9811d1bc4700 x1876350913787264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434039.078442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913787264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434039.078450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434039.078453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434039.078456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787264:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434039.974039:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434040.998073:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434042.022039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434043.046017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434044.070074:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434044.198187:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434044.198197:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434044.198202:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434044.198212:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6413 00000100:00080000:3.0:1789434044.198220:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434044.198230:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434044.198264:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913787392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434044.198295:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434044.198299:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434044.198301:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434044.198305:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6413 00000100:00080000:3.0:1789434044.198309:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434044.198314:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434044.198324:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434044.198339:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913787520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434044.198350:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434044.198352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913787392:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434044.198354:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434044.198359:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434044.198364:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6413 00000100:00080000:3.0:1789434044.198367:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434044.198371:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434044.198376:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434044.198388:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913787648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434044.198398:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434044.198401:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434044.198404:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434044.198408:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6413 00000100:00080000:3.0:1789434044.198412:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789434044.198417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434044.198418:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434044.198426:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434044.198439:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913787776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434044.198461:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434044/real 1789434044] req@ffff9811d1bc7100 x1876350913787392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434044.198480:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913787392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434044.198489:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434044.198493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434044.198496:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913787392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434044.198515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434044.198534:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434044.198545:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434044/real 1789434044] req@ffff9811d1bc4700 x1876350913787520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434044.198555:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913787520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434044.198562:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434044.198564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434044.198567:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434044.198571:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434044.198585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434044.198593:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434044/real 1789434044] req@ffff9811d1bc6300 x1876350913787648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434044.198602:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913787648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434044.198608:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434044.198610:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434044.198613:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434044.198619:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787776:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434044.198634:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434044.198643:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434044/real 1789434044] req@ffff9811d1bc5500 x1876350913787776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434044.198652:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913787776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434044.198661:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434044.198663:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434044.198666:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913787776:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434045.094030:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434046.118158:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434047.142213:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434048.166462:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434049.190051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434049.317906:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434049.317916:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434049.317920:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434049.317931:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6418 00000100:00080000:3.0:1789434049.317940:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434049.317952:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434049.317983:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913787904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434049.318003:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434049.318006:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434049.318007:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434049.318010:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6418 00000100:00080000:3.0:1789434049.318012:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434049.318015:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434049.318022:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434049.318030:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913788032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434049.318038:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434049.318039:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434049.318040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434049.318044:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6418 00000100:00100000:0.0:1789434049.318046:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913787904:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434049.318046:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434049.318050:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434049.318056:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434049.318077:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913788160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434049.318087:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434049.318091:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434049.318093:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434049.318097:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6418 00000100:00080000:3.0:1789434049.318102:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434049.318107:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434049.318111:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789434049.318123:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434049.318123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913788288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434049.318168:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434049/real 1789434049] req@ffff9811d1bc6680 x1876350913787904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434049.318186:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913787904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434049.318197:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434049.318201:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434049.318205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913787904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434049.318225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434049.318250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434049.318261:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434049/real 1789434049] req@ffff9811d1bc5c00 x1876350913788032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434049.318273:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913788032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434049.318281:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434049.318285:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434049.318288:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434049.318293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434049.318310:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434049.318319:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434049/real 1789434049] req@ffff9811d1bc5880 x1876350913788160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434049.318331:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913788160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434049.318338:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434049.318342:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434049.318345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434049.318350:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788288:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434049.318374:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434049.318384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434049/real 1789434049] req@ffff9811d1bc5f80 x1876350913788288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434049.318401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913788288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434049.318408:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434049.318411:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434049.318413:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788288:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434050.214029:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434051.237987:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434052.262872:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434053.286089:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434054.310158:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434054.438298:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434054.438307:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434054.438311:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434054.438320:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6424 00000100:00080000:3.0:1789434054.438331:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434054.438343:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434054.438381:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913788416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434054.438412:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434054.438415:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434054.438417:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434054.438421:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6424 00000100:00080000:3.0:1789434054.438425:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434054.438430:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434054.438439:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434054.438451:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913788544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434054.438460:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434054.438463:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434054.438464:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434054.438716:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913788416:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434054.438772:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434054.438809:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434054/real 1789434054] req@ffff9811d1bc4000 x1876350913788416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434054.438824:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913788416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434054.438924:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6424 00000100:00080000:3.0:1789434054.438929:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434054.438934:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434054.438939:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434054.438952:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913788672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434054.438963:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434054.438966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434054.438968:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434054.438971:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6424 00000100:00080000:3.0:1789434054.438974:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434054.439181:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434054.439187:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434054.439201:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913788800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434054.439369:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434054.439374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434054.439378:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913788416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434054.439385:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434054.439413:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434054.439427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434054/real 1789434054] req@ffff9811d1bc7800 x1876350913788544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434054.439445:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913788544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434054.439453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434054.439457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434054.439460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434054.440572:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434054.440612:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434054.440627:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434054/real 1789434054] req@ffff9811d1bc5180 x1876350913788672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434054.440641:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913788672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434054.440650:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434054.440653:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434054.440656:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434054.440663:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788800:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434054.440682:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434054.440692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434054/real 1789434054] req@ffff9811d1bc6a00 x1876350913788800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434054.440703:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913788800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434054.440717:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434054.440720:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434054.440723:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913788800:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434055.334011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434056.358074:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434057.383071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434058.406097:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434059.430115:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434059.557983:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434059.558006:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434059.558011:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434059.558022:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6429 00000100:00080000:1.0:1789434059.558032:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434059.558044:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434059.558083:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913788928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434059.558117:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434059.558121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434059.558123:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434059.558127:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6429 00000100:00080000:1.0:1789434059.558131:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434059.558137:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434059.558149:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434059.558163:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913789056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434059.558173:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434059.558183:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434059.558187:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434059.558189:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913788928:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434059.558195:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6429 00000100:00080000:1.0:1789434059.558201:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434059.558208:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434059.558213:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434059.558226:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913789184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434059.558239:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434059.558241:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434059.558243:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434059.558246:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6429 00000100:00080000:1.0:1789434059.558261:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434059.558265:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434059.558270:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434059.558279:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913789312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434059.558330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434059.558377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434059.558416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789312:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434060.454650:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434061.478067:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434062.502018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434062.631031:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434062.631079:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434062.631096:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434062.631139:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434062.631148:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434062.631155:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434062.631230:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434059/real 1789434062] req@ffff9811d0bdc700 x1876350913789056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434062.631253:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913789056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434062.631264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434062.631268:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434062.631273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434062.631281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434059/real 1789434062] req@ffff9811d0bdea00 x1876350913789184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434062.631293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913789184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434062.631300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434062.631303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434062.631305:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434062.631311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434059/real 1789434062] req@ffff9811d0bdd880 x1876350913789312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434062.631322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913789312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434062.631329:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434062.631332:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434062.631335:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789312:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789434062.631341:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434059/real 1789434062] req@ffff9811d0bdc000 x1876350913788928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434062.631351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913788928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434062.631357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434062.631360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434062.631363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913788928:192.168.203.128@tcp:250:kworker.0 00010000:00010000:2.0:1789434063.527950:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434064.550069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434064.678098:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434064.678117:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434064.678121:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434064.678131:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6434 00000100:00080000:3.0:1789434064.678139:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434064.678150:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434064.678182:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913789440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434064.678211:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434064.678215:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434064.678217:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434064.678221:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6434 00000100:00080000:3.0:1789434064.678224:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434064.678229:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434064.678239:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434064.678253:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913789568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434064.678262:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434064.678265:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434064.678267:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434064.678270:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6434 00000100:00080000:3.0:1789434064.678273:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434064.678277:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434064.678292:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434064.678304:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913789696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434064.678313:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434064.678316:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434064.678318:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434064.678321:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6434 00000100:00080000:3.0:1789434064.678323:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434064.678328:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434064.678332:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434064.678343:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913789824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434064.678366:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913789440:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434064.678425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434064.678460:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434064/real 1789434064] req@ffff9811d1bc7480 x1876350913789440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434064.678491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913789440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434064.678502:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434064.678506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434064.678510:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913789440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434064.678517:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434064.678537:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434064.678549:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434064/real 1789434064] req@ffff9811d1bc6a00 x1876350913789568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434064.678561:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913789568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434064.678570:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434064.678573:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434064.678576:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434064.678582:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434064.678600:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434064.678618:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434064/real 1789434064] req@ffff9811d1bc5180 x1876350913789696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434064.678637:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913789696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434064.678645:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434064.678649:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434064.678652:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434064.678658:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789824:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434064.678677:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434064.678688:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434064/real 1789434064] req@ffff9811d1bc7800 x1876350913789824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434064.678699:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913789824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434064.678707:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434064.678711:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434064.678714:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913789824:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434065.574066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434066.598059:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434067.621965:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434068.646036:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434069.670071:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434069.798056:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434069.798064:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434069.798067:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434069.798075:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6439 00000100:00080000:2.0:1789434069.798082:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434069.798091:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434069.798120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913789952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434069.798140:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434069.798143:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434069.798145:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434069.798153:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6439 00000100:00080000:2.0:1789434069.798156:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434069.798161:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434069.798170:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434069.798180:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913789952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434069.798185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913790080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434069.798194:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434069.798197:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434069.798198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434069.798201:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6439 00000100:00080000:2.0:1789434069.798205:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434069.798208:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434069.798220:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434069.798229:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913790208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434069.798235:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434069.798238:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434069.798241:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434069.798242:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434069.798245:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6439 00000100:00080000:2.0:1789434069.798248:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434069.798253:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434069.798257:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434069.798274:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913790336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434069.798276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434069/real 1789434069] req@ffff9811d2101180 x1876350913789952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434069.798302:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913789952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434069.798312:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434069.798315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434069.798319:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913789952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434069.798336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434069.798364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434069.798378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434069/real 1789434069] req@ffff9811d2101c00 x1876350913790080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434069.798391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913790080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434069.798399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434069.798402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434069.798405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434069.798411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434069.798429:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434069.798439:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434069/real 1789434069] req@ffff9811d2103b80 x1876350913790208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434069.798449:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913790208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434069.798461:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434069.798464:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434069.798466:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434069.798471:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790336:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434069.798490:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434069.798500:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434069/real 1789434069] req@ffff9811d2102a00 x1876350913790336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434069.798510:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913790336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434069.798518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434069.798521:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434069.798524:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790336:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434070.694063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434071.718423:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434072.742037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434073.766209:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434074.790063:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434074.919183:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434074.919193:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434074.919197:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434074.919206:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6444 00000100:00080000:3.0:1789434074.919215:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434074.919226:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434074.919261:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913790464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434074.919292:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434074.919296:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434074.919298:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434074.919302:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6444 00000100:00080000:3.0:1789434074.919306:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434074.919311:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434074.919319:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434074.919331:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913790592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434074.919340:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434074.919343:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434074.919345:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434074.919348:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6444 00000100:00080000:3.0:1789434074.919350:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434074.919354:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434074.919358:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434074.919367:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913790720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434074.919376:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434074.919379:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434074.919380:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434074.919383:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6444 00000100:00080000:3.0:1789434074.919386:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434074.919401:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434074.919405:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434074.919416:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913790848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434074.919508:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913790464:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434074.919564:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434074.919606:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434074/real 1789434074] req@ffff9811d1bc4000 x1876350913790464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434074.919622:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913790464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434074.919643:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434074.919647:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434074.919651:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913790464:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434074.919657:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434074.919679:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434074.919690:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434074/real 1789434074] req@ffff9811d1bc5f80 x1876350913790592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434074.919702:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913790592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434074.919709:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434074.919712:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434074.919722:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434074.919729:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434074.919745:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434074.919755:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434074/real 1789434074] req@ffff9811d1bc5880 x1876350913790720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434074.919766:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913790720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434074.919774:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434074.919777:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434074.919780:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434074.919786:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790848:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434074.919813:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434074.919824:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434074/real 1789434074] req@ffff9811d1bc5c00 x1876350913790848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434074.919871:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913790848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434074.919891:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434074.919895:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434074.919898:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913790848:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434075.814018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434076.838019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434077.862008:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434078.886022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434079.910015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434080.037946:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434080.037953:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434080.037956:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434080.037963:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6449 00000100:00080000:1.0:1789434080.037971:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434080.037980:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434080.038008:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913790976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434080.038031:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434080.038033:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434080.038034:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434080.038037:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6449 00000100:00080000:1.0:1789434080.038038:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434080.038041:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434080.038048:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434080.038056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913791104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434080.038062:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434080.038064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434080.038065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434080.038067:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6449 00000100:00080000:1.0:1789434080.038069:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434080.038071:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434080.038074:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434080.038079:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913790976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434080.038082:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913791232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434080.038096:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434080.038098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434080.038100:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434080.038103:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6449 00000100:00080000:1.0:1789434080.038107:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434080.038112:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434080.038116:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434080.038129:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913791360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434080.038149:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434080.038198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434080/real 1789434080] req@ffff9811d0bdf100 x1876350913790976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434080.038215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913790976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434080.038224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434080.038228:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434080.038232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913790976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434080.038239:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434080.038262:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434080.038274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434080/real 1789434080] req@ffff9811d0bde300 x1876350913791104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434080.038292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913791104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434080.038300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434080.038303:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434080.038306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434080.038420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434080.038443:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434080.038454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434080/real 1789434080] req@ffff9811d0bdf800 x1876350913791232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434080.038464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913791232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434080.038471:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434080.038475:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434080.038478:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434080.038483:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791360:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434080.038500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434080.038509:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434080/real 1789434080] req@ffff9811d0bdc000 x1876350913791360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434080.038525:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913791360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434080.038533:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434080.038535:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434080.038539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791360:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434080.934037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434081.958014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434082.982057:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434084.006015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434085.030083:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434085.158126:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434085.158140:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434085.158145:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434085.158158:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6454 00000100:00080000:1.0:1789434085.158171:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434085.158187:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434085.158234:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913791488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434085.158265:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434085.158268:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434085.158270:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434085.158273:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6454 00000100:00080000:1.0:1789434085.158276:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434085.158281:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434085.158290:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434085.158300:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913791616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434085.158309:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434085.158314:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434085.158316:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434085.158319:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6454 00000100:00080000:1.0:1789434085.158322:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434085.158326:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434085.158331:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434085.158341:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913791744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434085.158359:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434085.158363:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434085.158365:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434085.158369:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6454 00000100:00080000:1.0:1789434085.158373:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434085.158378:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434085.158383:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434085.158395:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913791872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434085.158400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913791488:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434085.158470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434085.158520:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434085/real 1789434085] req@ffff9811d0bdd880 x1876350913791488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434085.158539:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913791488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434085.158550:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434085.158555:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434085.158559:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913791488:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434085.158566:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434085.158586:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434085.158596:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434085/real 1789434085] req@ffff9811d0bdea00 x1876350913791616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434085.158614:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913791616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434085.158622:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434085.158626:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434085.158629:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434085.158635:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434085.158664:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434085.158672:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434085/real 1789434085] req@ffff9811d0bdc700 x1876350913791744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434085.158680:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913791744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434085.158685:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434085.158687:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434085.158690:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434085.158718:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791872:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434085.158738:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434085.158747:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434085/real 1789434085] req@ffff9811d0bded80 x1876350913791872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434085.158757:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913791872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434085.158771:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434085.158775:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434085.158778:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913791872:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434086.054005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434087.079006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434088.102134:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434089.126063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434090.150005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434090.278825:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434090.278853:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434090.278857:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434090.278867:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6459 00000100:00080000:2.0:1789434090.278876:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434090.278886:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434090.278921:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913792000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434090.278949:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434090.278953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434090.278954:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434090.278959:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6459 00000100:00080000:2.0:1789434090.278962:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434090.278966:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434090.278975:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434090.278988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913792128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434090.278998:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434090.279001:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434090.279002:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434090.279005:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6459 00000100:00080000:2.0:1789434090.279009:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434090.279013:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434090.279016:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434090.279025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913792256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434090.279034:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434090.279037:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434090.279039:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434090.279042:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6459 00000100:00080000:2.0:1789434090.279045:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434090.279049:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434090.279053:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434090.279062:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913792384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434090.279143:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913792000:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434090.279199:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434090.279247:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434090/real 1789434090] req@ffff9811d2101500 x1876350913792000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434090.279262:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913792000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434090.279270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434090.279274:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434090.279277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913792000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434090.279283:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434090.279302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434090.279313:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434090/real 1789434090] req@ffff9811d2100000 x1876350913792128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434090.279323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913792128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434090.279335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434090.279338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434090.279340:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434090.279345:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434090.279363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434090.279372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434090/real 1789434090] req@ffff9811d2103800 x1876350913792256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434090.279382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913792256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434090.279388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434090.279391:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434090.279394:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434090.279400:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792384:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434090.279417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434090.279427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434090/real 1789434090] req@ffff9811d2102d80 x1876350913792384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434090.279438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913792384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434090.279446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434090.279449:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434090.279452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792384:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434091.174029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434092.198079:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434093.222651:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434094.246194:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434095.270222:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434095.397951:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434095.397960:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434095.397964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434095.397974:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6464 00000100:00080000:2.0:1789434095.397982:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434095.397994:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434095.398055:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913792512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434095.398084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434095.398088:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434095.398090:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434095.398094:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6464 00000100:00080000:2.0:1789434095.398098:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434095.398103:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434095.398112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434095.398125:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913792640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434095.398134:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434095.398136:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434095.398138:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434095.398140:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6464 00000100:00080000:2.0:1789434095.398142:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434095.398147:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434095.398151:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434095.398166:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913792768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434095.398175:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434095.398179:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434095.398180:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434095.398183:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6464 00000100:00080000:2.0:1789434095.398186:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434095.398190:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434095.398195:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434095.398205:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913792896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434095.398325:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913792512:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434095.398382:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434095.398427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434095/real 1789434095] req@ffff9811d2102d80 x1876350913792512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434095.398447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913792512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434095.398457:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434095.398461:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434095.398465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913792512:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434095.398472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434095.398493:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434095.398506:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434095/real 1789434095] req@ffff9811d2103800 x1876350913792640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434095.398518:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913792640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434095.398526:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434095.398529:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434095.398532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434095.398540:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434095.398558:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434095.398569:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434095/real 1789434095] req@ffff9811d2100000 x1876350913792768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434095.398580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913792768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434095.398588:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434095.398591:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434095.398594:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434095.398600:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792896:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434095.398618:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434095.398629:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434095/real 1789434095] req@ffff9811d2101500 x1876350913792896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434095.398639:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913792896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434095.398647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434095.398649:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434095.398652:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913792896:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434096.293984:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434097.319231:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434098.342222:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434099.366098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434100.389998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434100.518721:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434100.518731:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434100.518735:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434100.518744:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6470 00000100:00080000:3.0:1789434100.518753:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434100.518763:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434100.518796:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913793024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434100.518825:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434100.518829:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434100.518852:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434100.518857:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6470 00000100:00080000:3.0:1789434100.518860:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434100.518865:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434100.518875:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434100.518887:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913793152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434100.518899:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434100.518902:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1789434100.518903:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913793024:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434100.518905:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434100.518913:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6470 00000100:00080000:3.0:1789434100.518916:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434100.518920:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434100.518924:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434100.518935:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913793280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434100.518945:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434100.518948:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434100.518950:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434100.518954:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6470 00000100:00080000:3.0:1789434100.518958:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434100.518964:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434100.518968:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789434100.518974:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434100.518980:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913793408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434100.519036:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434100/real 1789434100] req@ffff9811d1bc6680 x1876350913793024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434100.519054:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913793024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434100.519064:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434100.519068:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434100.519073:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913793024:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434100.519099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434100.519120:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434100.519132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434100/real 1789434100] req@ffff9811d1bc5500 x1876350913793152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434100.519142:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913793152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434100.519149:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434100.519152:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434100.519156:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434100.519161:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434100.519179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434100.519188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434100/real 1789434100] req@ffff9811d1bc6300 x1876350913793280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434100.519203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913793280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434100.519209:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434100.519211:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434100.519214:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434100.519218:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793408:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434100.519233:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434100.519242:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434100/real 1789434100] req@ffff9811d1bc4700 x1876350913793408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434100.519251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913793408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434100.519258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434100.519261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434100.519263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793408:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434101.414014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434102.437959:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434103.462019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434104.486155:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434105.510682:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434105.638485:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434105.638495:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434105.638499:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434105.638508:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6475 00000100:00080000:0.0:1789434105.638517:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434105.638529:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434105.638560:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913793536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434105.638588:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434105.638591:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434105.638593:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434105.638597:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6475 00000100:00080000:0.0:1789434105.638601:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434105.638607:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434105.638616:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434105.638629:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913793664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434105.638633:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913793536:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434105.638642:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434105.638649:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434105.638662:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434105.638665:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6475 00000100:00080000:0.0:1789434105.638668:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434105.638672:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434105.638681:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434105.638694:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913793792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434105.638705:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434105.638709:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434105.638711:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434105.638716:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6475 00000800:00000100:3.0:1789434105.638718:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434105.638721:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434105.638728:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434105.638734:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434105.638746:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913793920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434105.638756:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434105/real 1789434105] req@ffff9811c8cd5f80 x1876350913793536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434105.638771:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913793536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434105.638779:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434105.638782:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434105.638784:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913793536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434105.638801:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434105.638817:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434105.638825:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434105/real 1789434105] req@ffff9811c8cd6d80 x1876350913793664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434105.638849:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913793664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434105.638855:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434105.638857:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434105.638861:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434105.638866:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434105.638883:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434105.638894:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434105/real 1789434105] req@ffff9811c8cd5500 x1876350913793792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434105.638905:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913793792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434105.638912:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434105.638915:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434105.638917:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434105.638921:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434105.638935:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434105.638941:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434105/real 1789434105] req@ffff9811c8cd4a80 x1876350913793920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434105.638948:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913793920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434105.638952:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434105.638954:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434105.638956:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913793920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434106.534070:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434107.561428:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434108.582205:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434109.606249:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434110.630015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434110.758283:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434110.758297:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434110.758300:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434110.758318:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6480 00000100:00080000:2.0:1789434110.758326:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434110.758338:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434110.758372:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913794048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434110.758395:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434110.758399:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434110.758400:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434110.758405:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6480 00000100:00080000:2.0:1789434110.758408:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434110.758415:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434110.758425:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434110.758436:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913794048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434110.758440:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913794176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434110.758452:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434110.758455:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434110.758457:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434110.758460:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6480 00000100:00080000:2.0:1789434110.758463:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434110.758467:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434110.758472:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434110.758484:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913794304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434110.758494:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789434110.758507:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434110.758540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434110/real 1789434110] req@ffff9811d2102a00 x1876350913794048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434110.758557:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913794048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434110.758566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434110.758570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434110.758573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913794048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434110.758594:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434110.758630:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434110.758643:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434110/real 1789434110] req@ffff9811d2103b80 x1876350913794176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434110.758655:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913794176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434110.758663:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434110.758666:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434110.758669:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434110.758675:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434110.758701:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434110.758711:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434110/real 1789434110] req@ffff9811d2101c00 x1876350913794304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434110.758722:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913794304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434110.758729:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434110.758740:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434110.758743:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794304:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434110.758952:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434110.758955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434110.758959:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6480 00000100:00080000:2.0:1789434110.758963:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434110.758969:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434110.758974:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434110.758988:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913794432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434110.759036:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434110.759058:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434110.759078:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434110/real 1789434110] req@ffff9811d2101180 x1876350913794432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434110.759089:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913794432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434110.759097:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434110.759099:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434110.759102:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434111.654015:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434112.678028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434113.705690:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434114.726010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434115.750059:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434115.879146:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434115.879155:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434115.879160:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434115.879170:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6485 00000100:00080000:3.0:1789434115.879179:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434115.879190:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434115.879223:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913794560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434115.879253:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434115.879256:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434115.879258:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434115.879262:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6485 00000100:00080000:3.0:1789434115.879265:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434115.879269:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434115.879279:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434115.879294:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913794688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434115.879305:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434115.879309:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434115.879311:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434115.879320:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6485 00000100:00100000:0.0:1789434115.879324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913794560:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434115.879326:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434115.879335:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434115.879342:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434115.879355:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913794816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434115.879365:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434115.879368:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434115.879370:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434115.879377:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6485 00000100:00080000:3.0:1789434115.879381:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434115.879390:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434115.879396:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434115.879408:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913794944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434115.879422:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434115.879482:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434115/real 1789434115] req@ffff9811d1bc7100 x1876350913794560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434115.879505:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913794560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434115.879518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434115.879523:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434115.879528:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913794560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434115.879555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434115.879596:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434115.879610:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434115/real 1789434115] req@ffff9811d1bc4700 x1876350913794688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434115.879623:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913794688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434115.879632:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434115.879635:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434115.879638:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434115.879644:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434115.879664:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434115.879675:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434115/real 1789434115] req@ffff9811d1bc6300 x1876350913794816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434115.879686:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913794816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434115.879694:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434115.879697:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434115.879700:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434115.879706:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794944:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434115.879726:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434115.879735:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434115/real 1789434115] req@ffff9811d1bc5500 x1876350913794944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434115.879753:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913794944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434115.879762:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434115.879766:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434115.879769:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913794944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434116.774054:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434117.798620:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434118.822017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434119.846059:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434120.870017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434120.997961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434120.997971:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434120.997975:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434120.997985:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6490 00000100:00080000:0.0:1789434120.997997:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434120.998008:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434120.998044:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913795072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434120.998072:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434120.998075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434120.998077:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434120.998082:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6490 00000100:00080000:0.0:1789434120.998095:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434120.998101:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434120.998107:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913795072:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789434120.998112:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434120.998128:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913795200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434120.998140:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434120.998144:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434120.998146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434120.998150:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6490 00000100:00080000:0.0:1789434120.998155:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434120.998159:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434120.998164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789434120.998166:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434120.998178:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913795328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434120.998189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434120.998192:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434120.998194:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434120.998198:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6490 00000100:00080000:0.0:1789434120.998201:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434120.998206:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434120.998212:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434120.998218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434120/real 1789434120] req@ffff9811c8cd7b80 x1876350913795072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434120.998226:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913795456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434120.998243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913795072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434120.998255:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434120.998260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434120.998264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913795072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434120.998285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434120.998309:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434120.998322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434120/real 1789434120] req@ffff9811c8cd6300 x1876350913795200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434120.998334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913795200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434120.998348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434120.998351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434120.998354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434120.998360:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434120.998379:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434120.998390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434120/real 1789434120] req@ffff9811c8cd4000 x1876350913795328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434120.998401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913795328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434120.998408:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434120.998412:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434120.998414:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434120.998420:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434120.998440:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434120.998451:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434120/real 1789434120] req@ffff9811c8cd6a00 x1876350913795456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434120.998461:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913795456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434120.998469:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434120.998472:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434120.998475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434121.894009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434122.919555:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434123.942050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434124.966019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434125.990011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434126.118013:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434126.118021:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434126.118024:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434126.118034:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6495 00000100:00080000:2.0:1789434126.118041:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434126.118050:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434126.118092:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913795584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434126.118115:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434126.118118:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434126.118119:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434126.118122:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6495 00000100:00080000:2.0:1789434126.118125:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434126.118129:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434126.118136:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434126.118149:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913795712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434126.118161:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913795584:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434126.118170:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434126.118175:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434126.118177:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434126.118181:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6495 00000100:00080000:2.0:1789434126.118185:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434126.118191:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434126.118197:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434126.118212:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913795840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434126.118220:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434126.118223:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434126.118224:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434126.118227:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6495 00000100:00080000:2.0:1789434126.118230:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434126.118233:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434126.118237:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434126.118247:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913795968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434126.118257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434126.118286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434126.118306:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434127.014691:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434128.038051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434129.062010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434129.190199:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434129.190255:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434129.190288:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434129.190334:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434129.190343:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434129.190351:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789434129.190405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434126/real 1789434129] req@ffff9811d2101180 x1876350913795968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434129.190434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913795968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434129.190448:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434129.190454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434129.190459:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795968:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789434129.190467:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434126/real 1789434129] req@ffff9811d2101f80 x1876350913795712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434129.190492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913795712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434129.190500:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434129.190506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434129.190509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434129.190518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434126/real 1789434129] req@ffff9811d2100e00 x1876350913795840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434129.190529:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913795840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434129.190536:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434129.190539:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434129.190542:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913795840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434129.190549:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434126/real 1789434129] req@ffff9811d2101880 x1876350913795584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434129.190561:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913795584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434129.190569:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434129.190572:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434129.190575:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913795584:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789434130.086067:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434131.110035:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434131.238135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434131.238147:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434131.238151:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434131.238164:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6500 00000100:00080000:2.0:1789434131.238175:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434131.238188:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434131.238235:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913796096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434131.238273:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434131.238277:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434131.238280:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434131.238284:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6500 00000100:00080000:2.0:1789434131.238288:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434131.238294:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434131.238306:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434131.238322:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913796224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434131.238332:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434131.238336:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434131.238338:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434131.238342:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6500 00000100:00080000:2.0:1789434131.238346:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434131.238351:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434131.238356:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434131.238367:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913796352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434131.238380:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434131.238383:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434131.238385:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434131.238388:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6500 00000100:00080000:2.0:1789434131.238392:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434131.238396:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434131.238400:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434131.238410:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913796480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434131.238498:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913796096:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434131.238581:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434131.238627:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434131/real 1789434131] req@ffff9811d2101c00 x1876350913796096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434131.238645:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913796096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434131.238660:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434131.238664:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434131.238669:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913796096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434131.238676:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434131.238710:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434131.238730:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434131/real 1789434131] req@ffff9811d2103b80 x1876350913796224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434131.238742:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913796224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434131.238749:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434131.238753:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434131.238756:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434131.238762:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434131.239497:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434131.239519:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434131/real 1789434131] req@ffff9811d2102a00 x1876350913796352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434131.239533:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913796352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434131.239542:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434131.239545:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434131.239548:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434131.239556:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434131.239581:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434131.239593:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434131/real 1789434131] req@ffff9811d2101500 x1876350913796480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434131.239602:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913796480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434131.239609:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434131.239612:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434131.239614:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434132.134048:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434133.158035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434134.182001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434135.205994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434136.230038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434136.358003:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434136.358018:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434136.358023:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434136.358036:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6505 00000100:00080000:3.0:1789434136.358048:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434136.358063:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434136.358108:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913796608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434136.358142:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434136.358146:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434136.358152:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434136.358157:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6505 00000100:00080000:3.0:1789434136.358162:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434136.358166:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434136.358176:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434136.358187:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913796736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434136.358195:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434136.358198:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434136.358201:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434136.358203:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913796608:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434136.358205:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6505 00000100:00080000:3.0:1789434136.358210:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434136.358215:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434136.358218:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434136.358239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913796864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434136.358249:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434136.358253:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434136.358255:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434136.358259:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6505 00000100:00080000:3.0:1789434136.358263:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434136.358279:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434136.358285:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789434136.358287:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434136.358302:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913796992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434136.358336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434136/real 1789434136] req@ffff9811d1bc6680 x1876350913796608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434136.358366:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913796608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434136.358387:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434136.358392:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434136.358396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913796608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434136.358433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434136.358461:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434136.358481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434136/real 1789434136] req@ffff9811d1bc5c00 x1876350913796736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434136.358493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913796736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434136.358501:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434136.358504:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434136.358507:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434136.358513:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434136.358532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434136.358542:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434136/real 1789434136] req@ffff9811d1bc5880 x1876350913796864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434136.358554:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913796864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434136.358568:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434136.358571:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434136.358574:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434136.358580:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434136.358606:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434136.358617:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434136/real 1789434136] req@ffff9811d1bc5f80 x1876350913796992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434136.358628:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913796992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434136.358637:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434136.358640:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434136.358643:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913796992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434137.254036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434138.277983:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434139.301981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434140.326006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434141.350035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434141.477940:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434141.477952:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434141.477956:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434141.477966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6511 00000100:00080000:3.0:1789434141.477973:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434141.477984:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434141.478016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913797120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434141.478036:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434141.478039:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434141.478040:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434141.478042:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6511 00000100:00080000:3.0:1789434141.478045:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434141.478048:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434141.478062:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434141.478075:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913797248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434141.478082:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913797120:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434141.478089:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434141.478093:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434141.478095:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434141.478099:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6511 00000100:00080000:3.0:1789434141.478103:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434141.478108:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434141.478112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434141.478121:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913797376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434141.478135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434141.478140:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1789434141.478141:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434141.478142:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434141.478148:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6511 00000100:00080000:3.0:1789434141.478152:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434141.478158:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434141.478163:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434141.478175:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913797504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434141.478183:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434141/real 1789434141] req@ffff9811d1bc4000 x1876350913797120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434141.478203:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913797120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434141.478213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434141.478224:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434141.478228:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913797120:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434141.478248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434141.478268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434141.478279:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434141/real 1789434141] req@ffff9811d1bc7800 x1876350913797248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434141.478290:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913797248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434141.478298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434141.478301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434141.478304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434141.478309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434141.478324:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434141.478333:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434141/real 1789434141] req@ffff9811d1bc5180 x1876350913797376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434141.478344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913797376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434141.478350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434141.478353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434141.478356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434141.478361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797504:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434141.478377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434141.478396:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434141/real 1789434141] req@ffff9811d1bc6a00 x1876350913797504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434141.478409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913797504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434141.478416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434141.478418:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434141.478421:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434142.373996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434143.398012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434144.422010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434145.446108:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434146.470009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434146.598175:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434146.598185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434146.598189:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434146.598199:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6516 00000100:00080000:0.0:1789434146.598208:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434146.598218:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434146.598255:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913797632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434146.598278:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434146.598282:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434146.598284:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434146.598288:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6516 00000100:00080000:0.0:1789434146.598291:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434146.598297:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434146.598305:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434146.598316:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913797760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434146.598330:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434146.598333:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434146.598335:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434146.598338:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6516 00000100:00080000:0.0:1789434146.598340:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434146.598344:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434146.598358:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434146.598369:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913797888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434146.598386:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913797632:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434146.598431:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434146.598458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434146/real 1789434146] req@ffff9811c8cd5180 x1876350913797632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434146.598471:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913797632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434146.598478:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434146.598480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434146.598483:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913797632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434146.598498:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434146.598510:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434146.598517:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434146/real 1789434146] req@ffff9811c8cd7800 x1876350913797760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434146.598524:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913797760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434146.598529:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434146.598530:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434146.598532:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797760:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434146.598775:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434146.598779:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434146.598781:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434146.598785:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6516 00000100:00080000:0.0:1789434146.598789:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434146.598794:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434146.598801:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434146.598814:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913798016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434146.598898:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434146.598922:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434146.598934:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434146/real 1789434146] req@ffff9811c8cd7480 x1876350913797888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434146.598946:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913797888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434146.598954:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434146.598957:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434146.598960:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913797888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434146.598967:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434146.598986:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434146.598998:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434146/real 1789434146] req@ffff9811c8cd7100 x1876350913798016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434146.599008:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913798016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434146.599016:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434146.599019:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434146.599022:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434147.493988:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434148.518086:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434149.542044:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434150.565980:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434151.590085:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434151.717965:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434151.717973:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434151.717976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434151.717985:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6521 00000100:00080000:0.0:1789434151.717993:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434151.718002:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434151.718037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913798144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434151.718065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434151.718068:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434151.718070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434151.718074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6521 00000100:00080000:0.0:1789434151.718078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434151.718082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434151.718091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434151.718104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913798272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434151.718116:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789434151.718116:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913798144:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434151.718126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434151.718140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434151.718144:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6521 00000100:00080000:0.0:1789434151.718148:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434151.718153:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434151.718158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434151.718171:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913798400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434151.718176:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434151.718192:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434151.718196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434151.718198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434151.718203:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6521 00000100:00080000:0.0:1789434151.718207:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434151.718211:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434151.718216:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434151.718226:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913798528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434151.718231:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434151/real 1789434151] req@ffff9811c8cd6680 x1876350913798144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434151.718249:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913798144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434151.718257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434151.718260:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434151.718262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913798144:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434151.718288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434151.718303:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434151.718327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434151/real 1789434151] req@ffff9811c8cd4700 x1876350913798272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434151.718345:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913798272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434151.718353:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434151.718356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434151.718359:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434151.718376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434151.718395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434151.718405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434151/real 1789434151] req@ffff9811c8cd5c00 x1876350913798400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434151.718416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913798400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434151.718424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434151.718427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434151.718430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434151.718435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798528:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434151.718465:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434151.718481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434151/real 1789434151] req@ffff9811c8cd7100 x1876350913798528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434151.718493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913798528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434151.718505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434151.718509:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434151.718512:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434152.614005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434153.638010:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434154.662530:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434155.686059:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434156.710104:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434156.837974:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434156.837985:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434156.837990:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434156.838000:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6526 00000100:00080000:2.0:1789434156.838009:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434156.838021:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434156.838057:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913798656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434156.838081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434156.838085:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434156.838087:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434156.838091:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6526 00000100:00080000:2.0:1789434156.838095:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434156.838099:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434156.838109:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434156.838121:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913798784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434156.838129:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913798656:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434156.838135:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434156.838140:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434156.838142:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434156.838146:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6526 00000100:00080000:2.0:1789434156.838150:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434156.838154:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434156.838159:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434156.838171:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913798912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434156.838183:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434156.838186:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434156.838189:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789434156.838190:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434156.838197:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6526 00000100:00080000:2.0:1789434156.838203:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434156.838208:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434156.838214:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434156.838227:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913799040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434156.838236:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434156/real 1789434156] req@ffff9811d2100000 x1876350913798656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434156.838255:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913798656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434156.838265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434156.838269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434156.838285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913798656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434156.838307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434156.838329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434156.838341:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434156/real 1789434156] req@ffff9811d2103800 x1876350913798784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434156.838353:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913798784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434156.838361:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434156.838364:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434156.838368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434156.838374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434156.838390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434156.838400:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434156/real 1789434156] req@ffff9811d2102d80 x1876350913798912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434156.838411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913798912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434156.838419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434156.838422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434156.838425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913798912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434156.838430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434156.838447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434156.838458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434156/real 1789434156] req@ffff9811d2101500 x1876350913799040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434156.838468:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913799040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434156.838475:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434156.838478:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434156.838481:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434157.734058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434158.758013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434159.782178:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434160.806031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434161.830285:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434161.957937:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434161.957947:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434161.957952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434161.957962:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6531 00000100:00080000:0.0:1789434161.957970:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434161.957981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434161.958014:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913799168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434161.958038:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434161.958042:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434161.958044:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434161.958048:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6531 00000100:00080000:0.0:1789434161.958051:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434161.958056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434161.958065:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434161.958078:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913799296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434161.958088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434161.958092:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434161.958094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434161.958098:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6531 00000100:00080000:0.0:1789434161.958102:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789434161.958108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913799168:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434161.958109:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434161.958117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434161.958130:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913799424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434161.958141:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434161.958144:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434161.958146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434161.958160:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6531 00000100:00080000:0.0:1789434161.958168:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434161.958179:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789434161.958182:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789434161.958187:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434161.958203:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913799552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434161.958227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434161/real 1789434161] req@ffff9811c8cd7480 x1876350913799168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434161.958256:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913799168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434161.958266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434161.958270:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434161.958274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913799168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434161.958281:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434161.958302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434161.958314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434161/real 1789434161] req@ffff9811c8cd7800 x1876350913799296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434161.958325:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913799296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434161.958332:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434161.958334:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434161.958337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434161.958359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434161.958376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434161.958385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434161/real 1789434161] req@ffff9811c8cd5180 x1876350913799424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434161.958402:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913799424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434161.958409:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434161.958413:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434161.958416:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434161.958421:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434161.958438:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434161.958448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434161/real 1789434161] req@ffff9811c8cd6a00 x1876350913799552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434161.958458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913799552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434161.958466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434161.958469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434161.958472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434162.853999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434163.878140:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434164.902082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434165.926612:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434166.951724:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434167.081067:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434167.081082:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434167.081088:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434167.081100:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6536 00000100:00080000:3.0:1789434167.081111:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434167.081124:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434167.081174:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913799680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434167.081646:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434167.081652:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434167.081654:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434167.081660:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6536 00000100:00080000:3.0:1789434167.081664:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434167.081671:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434167.081682:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434167.081699:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913799808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434167.081709:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434167.081712:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434167.081714:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434167.081717:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6536 00000100:00080000:3.0:1789434167.081721:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434167.081726:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434167.081731:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434167.081742:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913799936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434167.081752:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434167.081756:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434167.081757:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434167.081762:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6536 00000100:00080000:3.0:1789434167.081765:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434167.081769:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434167.081774:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434167.081785:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913800064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434167.082048:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913799680:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434167.082124:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434167.082188:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434167/real 1789434167] req@ffff9811d1bc7480 x1876350913799680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434167.082231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913799680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434167.082243:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434167.082248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434167.082252:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913799680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434167.082260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434167.082283:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434167.082296:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434167/real 1789434167] req@ffff9811d1bc6a00 x1876350913799808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434167.082309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913799808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434167.082319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434167.082322:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434167.082326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434167.082332:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434167.082349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434167.082359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434167/real 1789434167] req@ffff9811d1bc5180 x1876350913799936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434167.082371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913799936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434167.082385:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434167.082389:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434167.082392:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913799936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434167.082398:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800064:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434167.082417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434167.082428:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434167/real 1789434167] req@ffff9811d1bc7800 x1876350913800064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434167.082440:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913800064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434167.082448:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434167.082451:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434167.082454:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434167.974114:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434168.999074:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434170.022140:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434171.046017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434172.070127:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434172.197911:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434172.197921:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434172.197925:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434172.197936:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6541 00000100:00080000:3.0:1789434172.197954:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434172.197966:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434172.198008:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913800192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434172.198038:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434172.198042:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434172.198044:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434172.198048:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6541 00000100:00080000:3.0:1789434172.198052:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434172.198056:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434172.198066:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434172.198078:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913800320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434172.198094:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434172.198097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434172.198099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434172.198129:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6541 00000100:00080000:3.0:1789434172.198133:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434172.198137:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434172.198142:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434172.198209:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913800192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434172.198286:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913800448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434172.198298:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434172.198303:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434172.198305:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434172.198308:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6541 00000100:00080000:3.0:1789434172.198312:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434172.198317:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434172.198322:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434172.198336:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913800576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434172.198396:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434172.198471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434172/real 1789434172] req@ffff9811d1bc4000 x1876350913800192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434172.198495:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913800192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434172.198512:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434172.198518:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434172.198523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913800192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434172.198533:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434172.198568:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434172.198582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434172/real 1789434172] req@ffff9811d1bc5f80 x1876350913800320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434172.198595:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913800320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434172.198604:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434172.198608:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434172.198612:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434172.198646:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434172.198667:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434172.198678:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434172/real 1789434172] req@ffff9811d1bc5880 x1876350913800448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434172.198689:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913800448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434172.198696:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434172.198700:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434172.198703:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434172.198709:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434172.198735:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434172.198745:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434172/real 1789434172] req@ffff9811d1bc5c00 x1876350913800576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434172.198757:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913800576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434172.198766:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434172.198769:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434172.198772:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434173.094330:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434174.118276:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434175.142006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434176.166037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434177.190119:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434177.317980:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434177.317989:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434177.317995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434177.318005:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6546 00000100:00080000:3.0:1789434177.318015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434177.318026:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434177.318061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913800704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434177.318087:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434177.318091:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434177.318092:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434177.318096:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6546 00000100:00080000:3.0:1789434177.318100:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434177.318105:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434177.318114:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434177.318138:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913800832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434177.318147:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434177.318150:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434177.318152:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434177.318156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6546 00000100:00080000:3.0:1789434177.318159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434177.318164:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434177.318169:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434177.318180:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913800960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434177.318581:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434177.318587:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434177.318590:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434177.318596:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6546 00000100:00080000:3.0:1789434177.318601:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434177.318607:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434177.318612:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913800704:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789434177.318615:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434177.318635:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913801088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434177.318680:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434177.318724:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434177/real 1789434177] req@ffff9811d1bc6680 x1876350913800704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434177.318741:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913800704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434177.318751:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434177.318755:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434177.318759:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913800704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434177.318765:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434177.318783:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434177.318794:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434177/real 1789434177] req@ffff9811d1bc5500 x1876350913800832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434177.318805:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913800832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434177.318818:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434177.318821:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434177.318824:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434177.318829:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434177.318878:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434177.318888:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434177/real 1789434177] req@ffff9811d1bc6300 x1876350913800960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434177.318898:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913800960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434177.318905:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434177.318907:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434177.318910:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913800960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434177.318932:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434177.318950:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434177.318959:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434177/real 1789434177] req@ffff9811d1bc4700 x1876350913801088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434177.318970:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913801088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434177.318978:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434177.318981:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434177.318984:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434178.215174:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434179.238355:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434180.262138:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434181.286086:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434182.310691:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434182.437951:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434182.438021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434182.438025:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434182.438035:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6552 00000100:00080000:3.0:1789434182.438044:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434182.438055:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434182.438104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913801216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434182.438185:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913801216:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434182.438244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434182.438630:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434182.438635:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434182.438637:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434182.438641:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6552 00000100:00080000:3.0:1789434182.438646:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434182.438651:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434182.438660:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434182.438675:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913801344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434182.438894:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434182/real 1789434182] req@ffff9811d1bc7100 x1876350913801216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434182.438912:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913801216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434182.438922:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434182.438926:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434182.438929:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913801216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434182.438950:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434182.438985:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434182.438997:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434182/real 1789434182] req@ffff9811d1bc4700 x1876350913801344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434182.439008:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913801344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434182.439016:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434182.439019:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434182.439021:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801344:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434182.439894:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434182.439904:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434182.439908:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434182.439918:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6552 00000100:00080000:3.0:1789434182.439939:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434182.439954:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434182.439974:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434182.440007:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913801472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434182.440029:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434182.440036:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434182.440038:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434182.440042:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6552 00000100:00080000:3.0:1789434182.440046:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434182.440052:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434182.440057:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434182.440068:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434182.440072:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913801600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434182.440153:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434182.440182:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434182/real 1789434182] req@ffff9811d1bc6300 x1876350913801472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434182.440199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913801472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434182.440209:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434182.440213:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434182.440217:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434182.440249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801600:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434182.440282:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434182.440294:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434182/real 1789434182] req@ffff9811d1bc5500 x1876350913801600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434182.440305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913801600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434182.440314:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434182.440316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434182.440318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434183.334123:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434184.358017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434185.382007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434186.406421:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434187.430307:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434187.557968:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434187.557978:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434187.557983:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434187.557994:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6557 00000100:00080000:3.0:1789434187.558005:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434187.558016:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434187.558068:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913801728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434187.558098:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434187.558102:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434187.558104:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434187.558109:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6557 00000100:00080000:3.0:1789434187.558113:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434187.558120:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434187.558144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434187.558155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913801856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434187.558163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434187.558165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434187.558167:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434187.558170:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6557 00000100:00080000:3.0:1789434187.558172:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434187.558176:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434187.558179:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434187.558185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913801984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434187.558191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434187.558193:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434187.558194:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434187.558196:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6557 00000100:00080000:3.0:1789434187.558197:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434187.558199:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434187.558201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434187.558206:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913802112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434187.559398:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913801728:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434187.559500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434187.559556:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434187/real 1789434187] req@ffff9811d1bc6680 x1876350913801728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434187.559580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913801728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434187.559591:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434187.559596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434187.559600:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913801728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434187.559608:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434187.559634:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434187.559648:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434187/real 1789434187] req@ffff9811d1bc5c00 x1876350913801856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434187.559660:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913801856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434187.559668:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434187.559671:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434187.559674:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434187.559680:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434187.559698:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434187.559708:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434187/real 1789434187] req@ffff9811d1bc5880 x1876350913801984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434187.559720:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913801984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434187.559732:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434187.559736:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434187.559739:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913801984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434187.559745:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434187.559764:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434187.559775:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434187/real 1789434187] req@ffff9811d1bc5f80 x1876350913802112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434187.559786:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913802112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434187.559795:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434187.559799:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434187.559802:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434188.454198:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434189.478001:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434190.502219:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434191.525997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434192.550052:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434192.678312:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434192.678322:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434192.678421:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434192.678432:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6562 00000100:00080000:0.0:1789434192.678443:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434192.678454:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434192.678490:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913802240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434192.678522:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434192.678526:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434192.678529:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434192.678533:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6562 00000100:00080000:0.0:1789434192.678537:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434192.678541:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434192.678551:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434192.678563:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913802368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434192.678571:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434192.678574:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434192.678576:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434192.678580:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6562 00000100:00080000:0.0:1789434192.678583:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434192.678587:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434192.678592:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434192.678607:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913802496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434192.678616:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434192.678619:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434192.678621:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434192.678624:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6562 00000100:00080000:0.0:1789434192.678628:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434192.678632:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434192.678637:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434192.678647:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913802624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434192.678720:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913802240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434192.678807:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434192.678829:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434192.678868:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434193.573977:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434194.598117:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434195.622065:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789434195.750174:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789434195.750220:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789434195.750233:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434195.750265:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434195.750271:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434195.750275:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789434195.750315:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434192/real 1789434195] req@ffff9811c8cd4000 x1876350913802240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434195.750344:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913802240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434195.750355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434195.750360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434195.750365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913802240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434195.750372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434192/real 1789434195] req@ffff9811c8cd6300 x1876350913802368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434195.750383:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913802368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434195.750391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434195.750394:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434195.750397:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434195.750402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434192/real 1789434195] req@ffff9811c8cd7b80 x1876350913802496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434195.750413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913802496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434195.750420:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434195.750423:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434195.750436:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434195.750444:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434192/real 1789434195] req@ffff9811c8cd4a80 x1876350913802624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434195.750458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913802624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434195.750465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434195.750468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434195.750471:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434196.646047:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434197.670305:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434197.798120:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434197.798142:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434197.798146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434197.798157:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6567 00000100:00080000:3.0:1789434197.798168:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434197.798181:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434197.798215:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913802752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434197.798282:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913802752:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434197.798344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434197.798441:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434197.798446:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434197.798448:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434197.798452:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6567 00000100:00080000:3.0:1789434197.798456:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434197.798462:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434197.798473:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434197.798487:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913802880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434197.798553:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434197/real 1789434197] req@ffff9811d1bc4000 x1876350913802752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434197.798574:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913802752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434197.798586:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434197.798590:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434197.798593:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913802752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434197.798620:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434197.798642:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434197.798653:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434197/real 1789434197] req@ffff9811d1bc7800 x1876350913802880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434197.798664:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913802880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434197.798672:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434197.798675:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434197.798678:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913802880:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434197.798943:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434197.798947:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434197.798949:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434197.798954:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6567 00000100:00080000:3.0:1789434197.798958:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434197.798963:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434197.798969:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434197.798993:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913803008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434197.799004:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434197.799007:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434197.799009:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434197.799013:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6567 00000100:00080000:3.0:1789434197.799016:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434197.799020:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434197.799026:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434197.799036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913803136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434197.799084:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434197.799109:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434197.799121:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434197/real 1789434197] req@ffff9811d1bc5180 x1876350913803008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434197.799133:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913803008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434197.799140:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434197.799143:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434197.799147:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434197.799153:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434197.799172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434197.799182:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434197/real 1789434197] req@ffff9811d1bc6a00 x1876350913803136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434197.799193:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913803136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434197.799207:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434197.799210:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434197.799213:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434198.694521:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434199.718092:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434200.742053:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434201.766079:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434202.790046:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434202.917912:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434202.917924:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434202.917929:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434202.917947:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6572 00000100:00080000:3.0:1789434202.917957:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434202.917968:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434202.918008:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913803264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434202.918035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434202.918038:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434202.918040:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434202.918054:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6572 00000100:00080000:3.0:1789434202.918058:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434202.918062:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434202.918072:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434202.918086:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913803392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434202.918096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434202.918099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434202.918100:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434202.918103:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6572 00000100:00080000:3.0:1789434202.918106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434202.918112:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434202.918117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434202.918129:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913803520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434202.918139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434202.918143:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434202.918145:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434202.918150:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6572 00000100:00080000:3.0:1789434202.918154:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434202.918159:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434202.918164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434202.918174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913803264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434202.918184:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913803648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434202.918270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434202.918327:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434202/real 1789434202] req@ffff9811d1bc7480 x1876350913803264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434202.918343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913803264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434202.918352:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434202.918356:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434202.918358:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913803264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434202.918365:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434202.918382:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434202.918390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434202/real 1789434202] req@ffff9811d1bc6a00 x1876350913803392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434202.918397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913803392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434202.918402:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434202.918405:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434202.918407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434202.918411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434202.918421:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434202.918427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434202/real 1789434202] req@ffff9811d1bc5180 x1876350913803520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434202.918434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913803520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434202.918438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434202.918440:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434202.918441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434202.918463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434202.918474:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434202.918479:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434202/real 1789434202] req@ffff9811d1bc7800 x1876350913803648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434202.918486:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913803648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434202.918491:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434202.918492:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434202.918503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434203.814018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434204.838036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434205.862065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434206.886076:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434207.910529:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434208.037947:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434208.037959:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434208.037964:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434208.037974:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6577 00000100:00080000:3.0:1789434208.037982:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434208.037992:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434208.038024:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913803776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434208.038052:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434208.038056:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434208.038057:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434208.038061:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6577 00000100:00080000:3.0:1789434208.038064:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434208.038067:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434208.038077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434208.038087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913803904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434208.038097:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434208.038100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434208.038101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434208.038105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6577 00000100:00080000:3.0:1789434208.038107:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434208.038111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434208.038116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434208.038126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913804032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434208.038136:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434208.038139:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434208.038141:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434208.038144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6577 00000100:00080000:3.0:1789434208.038147:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434208.038153:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434208.038162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913803776:192.168.203.128@tcp:250:kworker.0 00000008:00100000:3.0:1789434208.038163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434208.038184:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913804160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434208.038287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434208.038352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434208/real 1789434208] req@ffff9811d1bc4000 x1876350913803776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434208.038374:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913803776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434208.038386:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434208.038390:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434208.038395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913803776:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434208.038404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434208.038432:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434208.038446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434208/real 1789434208] req@ffff9811d1bc5f80 x1876350913803904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434208.038458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913803904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434208.038466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434208.038470:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434208.038473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913803904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434208.038479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434208.038506:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434208.038517:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434208/real 1789434208] req@ffff9811d1bc5880 x1876350913804032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434208.038535:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913804032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434208.038541:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434208.038544:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434208.038547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434208.038572:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804160:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434208.038592:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434208.038604:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434208/real 1789434208] req@ffff9811d1bc5c00 x1876350913804160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434208.038615:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913804160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434208.038624:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434208.038627:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434208.038630:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434208.934103:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434209.958092:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434210.983031:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434212.006077:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434213.030572:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434213.158669:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434213.158682:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434213.158687:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434213.158699:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6582 00000100:00080000:3.0:1789434213.158710:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434213.158724:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434213.158772:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913804288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434213.158810:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434213.158814:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434213.158816:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434213.158820:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6582 00000100:00080000:3.0:1789434213.158824:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434213.158829:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434213.158882:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434213.158895:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913804416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434213.158905:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434213.158909:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434213.158911:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434213.158915:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6582 00000100:00080000:3.0:1789434213.158918:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434213.158923:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434213.158927:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434213.158937:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913804544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434213.158946:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434213.158949:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434213.158951:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434213.158954:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6582 00000100:00080000:3.0:1789434213.158957:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434213.158961:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434213.158965:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434213.158976:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913804672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434213.159075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913804288:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434213.159151:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434213.159199:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434213/real 1789434213] req@ffff9811d1bc6680 x1876350913804288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434213.159219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913804288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434213.159232:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434213.159236:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434213.159240:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913804288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434213.159247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434213.159274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434213.159287:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434213/real 1789434213] req@ffff9811d1bc5500 x1876350913804416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434213.159298:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913804416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434213.159307:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434213.159310:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434213.159313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434213.159318:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434213.159336:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434213.159346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434213/real 1789434213] req@ffff9811d1bc6300 x1876350913804544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434213.159355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913804544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434213.159362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434213.159365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434213.159368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434213.159374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434213.159392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434213.159402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434213/real 1789434213] req@ffff9811d1bc4700 x1876350913804672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434213.159414:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913804672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434213.159422:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434213.159425:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434213.159427:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434214.055139:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434215.080285:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434216.102182:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434217.126274:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434218.150217:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434218.281334:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434218.281344:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434218.281348:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434218.281359:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6587 00000100:00080000:1.0:1789434218.281369:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434218.281381:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434218.281415:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913804800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434218.281442:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434218.281445:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434218.281447:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434218.281451:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6587 00000100:00080000:1.0:1789434218.281455:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434218.281459:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434218.281470:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434218.281481:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913804928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434218.281491:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434218.281493:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913804800:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434218.281494:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434218.281498:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434218.281505:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6587 00000100:00080000:1.0:1789434218.281508:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434218.281512:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434218.281517:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434218.281528:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913805056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434218.281538:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434218.281541:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434218.281543:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434218.281547:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6587 00000100:00080000:1.0:1789434218.281551:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434218.281557:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434218.281559:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789434218.281562:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434218.281574:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913805184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434218.281614:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434218/real 1789434218] req@ffff9811d0bdfb80 x1876350913804800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434218.281633:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913804800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434218.281642:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434218.281646:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434218.281649:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913804800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434218.281670:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434218.281693:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434218.281703:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434218/real 1789434218] req@ffff9811d0bdf480 x1876350913804928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434218.281714:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913804928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434218.281722:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434218.281726:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434218.281729:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913804928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434218.281734:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434218.281751:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434218.281760:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434218/real 1789434218] req@ffff9811d0bde680 x1876350913805056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434218.281785:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913805056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434218.281792:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434218.281796:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434218.281799:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434218.281804:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805184:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434218.281878:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434218.281890:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434218/real 1789434218] req@ffff9811d0bdc380 x1876350913805184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434218.281907:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913805184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434218.281915:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434218.281918:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434218.281921:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434219.174177:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434220.198095:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434221.222453:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434222.247891:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434223.270081:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434223.398220:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434223.398233:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434223.398239:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434223.398252:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6592 00000100:00080000:0.0:1789434223.398263:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434223.398277:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434223.398326:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913805312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434223.398410:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913805312:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434223.398485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434223.398525:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434223.398529:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434223.398532:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434223.398536:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6592 00000100:00080000:0.0:1789434223.398540:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434223.398545:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434223.398556:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434223.398570:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913805440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434223.398626:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434223/real 1789434223] req@ffff9811c8cd5500 x1876350913805312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434223.398645:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913805312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434223.398666:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434223.398671:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434223.398675:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913805312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434223.398758:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805440:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434223.399540:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789434223.399551:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434223.399556:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434223.399560:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434223.399573:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6592 00000100:00100000:3.0:1789434223.399575:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434223/real 1789434223] req@ffff9811c8cd6d80 x1876350913805440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434223.399581:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434223.399593:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434223.399597:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913805440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789434223.399602:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789434223.399612:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434223.399617:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434223.399622:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434223.399625:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913805568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434223.399640:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434223.399644:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1789434223.399646:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805568:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434223.399647:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434223.399656:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6592 00000100:00080000:0.0:1789434223.399661:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434223.399667:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434223.399673:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:3.0:1789434223.399678:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434223.399689:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913805696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434223.399693:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434223/real 1789434223] req@ffff9811c8cd5f80 x1876350913805568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434223.400546:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913805568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434223.400555:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434223.400559:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434223.400563:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434223.400579:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434223.400607:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434223.400619:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434223/real 1789434223] req@ffff9811c8cd6d80 x1876350913805696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434223.400631:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913805696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434223.400639:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434223.400642:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434223.400645:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434224.294242:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434225.318294:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434226.342820:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434227.366080:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434228.390292:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434228.518031:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434228.518042:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434228.518047:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434228.518057:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6598 00000100:00080000:1.0:1789434228.518065:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434228.518076:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434228.518110:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913805824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434228.518141:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434228.518152:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434228.518154:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434228.518159:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6598 00000100:00080000:1.0:1789434228.518162:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434228.518172:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434228.518182:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434228.518194:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913805952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434228.518204:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434228.518207:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434228.518208:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434228.518212:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6598 00000100:00080000:1.0:1789434228.518215:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434228.518219:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434228.518223:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434228.518233:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913806080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434228.518242:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434228.518245:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434228.518247:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434228.518250:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6598 00000100:00080000:1.0:1789434228.518253:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434228.518257:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434228.518261:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434228.518271:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913806208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434228.520828:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913805824:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434228.520926:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434228.520967:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434228/real 1789434228] req@ffff9811d0bddc00 x1876350913805824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434228.520988:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913805824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434228.520999:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434228.521003:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434228.521006:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913805824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434228.521014:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434228.521039:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434228.521051:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434228/real 1789434228] req@ffff9811d0bdd500 x1876350913805952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434228.521062:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913805952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434228.521071:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434228.521075:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434228.521079:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913805952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434228.521085:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434228.521103:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434228.521113:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434228/real 1789434228] req@ffff9811d0bdd180 x1876350913806080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434228.521124:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913806080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434228.521131:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434228.521135:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434228.521138:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434228.521144:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434228.521165:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434228.521177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434228/real 1789434228] req@ffff9811d0bdc380 x1876350913806208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434228.521188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913806208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434228.521195:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434228.521198:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434228.521202:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434229.414780:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434230.438134:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434231.462767:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434232.486066:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434233.510464:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434233.638169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434233.638179:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434233.638183:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434233.638192:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6603 00000100:00080000:2.0:1789434233.638200:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434233.638212:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434233.638248:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913806336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434233.638336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913806336:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434233.638373:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434233.638378:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434233.638380:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434233.638386:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6603 00000100:00080000:2.0:1789434233.638390:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789434233.638395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434233.638398:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434233.638410:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434233.638425:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913806464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434233.638436:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789434233.638439:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434233/real 1789434233] req@ffff9811d2102a00 x1876350913806336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789434233.638440:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434233.638444:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434233.638452:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6603 00000100:00080000:2.0:1789434233.638459:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434233.638465:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434233.638470:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913806336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789434233.638472:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789434233.638486:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789434233.638490:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913806592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434233.638494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434233.638503:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913806336:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434233.638509:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434233.638513:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434233.638515:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434233.638520:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6603 00000100:00080000:2.0:1789434233.638524:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789434233.638530:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806464:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434233.638531:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434233.638539:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434233.638554:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913806720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434233.638563:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434233.638582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434233/real 1789434233] req@ffff9811d2103b80 x1876350913806464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434233.638595:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913806464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434233.638603:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434233.638606:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434233.638610:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434233.638615:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434233.638635:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434233.638645:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434233/real 1789434233] req@ffff9811d2101c00 x1876350913806592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434233.638656:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913806592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434233.638664:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434233.638667:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434233.638670:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434233.638683:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806720:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434233.638702:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434233.638729:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434233/real 1789434233] req@ffff9811d2101880 x1876350913806720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434233.638740:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913806720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434233.638749:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434233.638751:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434233.638754:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434234.534064:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434235.558880:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434236.583646:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434237.606062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434238.632160:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434238.759544:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434238.759555:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434238.759560:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434238.759569:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6608 00000100:00080000:2.0:1789434238.759578:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434238.759589:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434238.759626:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913806848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434238.759698:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913806848:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434238.759754:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434238.759785:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434238.759789:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434238.759791:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434238.759795:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6608 00000100:00080000:2.0:1789434238.759799:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434238.759804:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434238.759814:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434238.759826:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913806976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434238.760138:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434238.760143:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434238.760145:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434238.760148:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6608 00000100:00080000:2.0:1789434238.760153:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434238.760158:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434238.760163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434238.760175:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913807104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434238.760185:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434238.760188:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434238.760189:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434238.760192:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6608 00000100:00080000:2.0:1789434238.760195:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434238.760199:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434238.760203:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434238.760212:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913807232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434238.760610:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434238/real 1789434238] req@ffff9811d2100e00 x1876350913806848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434238.760702:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913806848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434238.760712:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434238.760716:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434238.760720:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913806848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434238.760745:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434238.760772:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434238.760787:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434238/real 1789434238] req@ffff9811d2101f80 x1876350913806976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434238.760798:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913806976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434238.760806:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434238.760810:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434238.760813:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913806976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434238.760818:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434238.760858:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434238.760869:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434238/real 1789434238] req@ffff9811d2101180 x1876350913807104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434238.760879:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913807104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434238.760891:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434238.760895:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434238.760898:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434238.760903:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434238.760924:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434238.760934:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434238/real 1789434238] req@ffff9811d2101880 x1876350913807232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434238.760945:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913807232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434238.760953:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434238.760956:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434238.760959:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434239.654374:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434240.678715:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434241.702024:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434242.726471:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434243.750493:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434243.878045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434243.878055:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434243.878059:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434243.878069:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6613 00000100:00080000:2.0:1789434243.878079:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434243.878090:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434243.878127:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913807360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434243.878155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434243.878158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434243.878160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434243.878163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6613 00000100:00080000:2.0:1789434243.878166:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434243.878170:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434243.878178:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434243.878190:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913807488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434243.878199:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434243.878202:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434243.878204:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434243.878207:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6613 00000100:00080000:2.0:1789434243.878210:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434243.878213:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434243.878217:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434243.878225:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913807616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434243.878243:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434243.878246:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434243.878247:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434243.878250:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6613 00000100:00080000:2.0:1789434243.878252:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434243.878256:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434243.878260:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434243.878269:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913807744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434243.878715:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913807360:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434243.878780:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434243.878816:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434243/real 1789434243] req@ffff9811d2101c00 x1876350913807360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434243.878960:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913807360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434243.878970:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434243.878974:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434243.878977:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913807360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434243.878984:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434243.879005:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434243.879017:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434243/real 1789434243] req@ffff9811d2103b80 x1876350913807488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434243.879029:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913807488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434243.879036:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434243.879038:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434243.879041:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434243.879046:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434243.879060:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434243.879068:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434243/real 1789434243] req@ffff9811d2102a00 x1876350913807616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434243.879079:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913807616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434243.879086:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434243.879088:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434243.879091:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434243.879097:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434243.879113:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434243.879121:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434243/real 1789434243] req@ffff9811d2101500 x1876350913807744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434243.879131:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913807744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434243.879137:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434243.879140:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434243.879143:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913807744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434244.774060:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434245.799671:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434246.822661:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434247.846222:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434248.871007:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434248.997980:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434248.997993:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434248.997997:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434248.998008:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6618 00000100:00080000:2.0:1789434248.998019:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434248.998032:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434248.998071:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913807872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434248.998152:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913807872:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434248.998214:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434248.998247:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434248.998250:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434248.998252:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434248.998256:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6618 00000100:00080000:2.0:1789434248.998261:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434248.998266:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434248.998277:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434248.998289:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913808000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434248.998346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434248/real 1789434248] req@ffff9811d2102d80 x1876350913807872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434248.998363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913807872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434248.998373:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434248.998377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434248.998381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913807872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434248.998405:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434248.998440:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434248.998454:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434248/real 1789434248] req@ffff9811d2103800 x1876350913808000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434248.998471:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913808000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434248.998479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434248.998482:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434248.998485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808000:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434248.998624:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434248.998628:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434248.998630:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434248.998634:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6618 00000100:00080000:2.0:1789434248.998638:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434248.998643:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434248.998649:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434248.998661:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913808128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434248.998710:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434248.998738:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434248.998759:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434248.998762:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434248.998764:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434248.998767:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6618 00000100:00080000:2.0:1789434248.998770:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434248.998775:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434248.998780:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434248.998793:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913808256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434248.998940:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434248/real 1789434248] req@ffff9811d2100000 x1876350913808128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434248.998953:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913808128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434248.998960:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434248.998963:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434248.998966:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434248.998978:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808256:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434248.998997:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434248.999007:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434248/real 1789434248] req@ffff9811d2103800 x1876350913808256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434248.999015:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913808256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434248.999028:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434248.999031:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434248.999034:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434249.894205:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434250.918043:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434251.942039:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434252.966098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434253.990406:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434254.118372:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434254.118382:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434254.118386:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434254.118396:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6623 00000100:00080000:3.0:1789434254.118406:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434254.118417:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434254.118466:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913808384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434254.118495:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434254.118498:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434254.118500:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434254.118504:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6623 00000100:00080000:3.0:1789434254.118507:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434254.118512:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434254.118521:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434254.118534:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913808512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434254.118543:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434254.118546:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434254.118548:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434254.118551:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6623 00000100:00080000:3.0:1789434254.118565:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434254.118570:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434254.118574:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434254.118585:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913808640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434254.118595:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434254.118598:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434254.118599:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434254.118603:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6623 00000100:00080000:3.0:1789434254.118606:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434254.118609:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434254.118613:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434254.118622:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913808768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434254.119989:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913808384:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434254.120078:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434254.120138:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434254/real 1789434254] req@ffff9811d1bc7100 x1876350913808384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434254.120158:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913808384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434254.120170:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434254.120174:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434254.120178:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913808384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434254.120186:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434254.120210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434254.120222:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434254/real 1789434254] req@ffff9811d1bc4700 x1876350913808512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434254.120233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913808512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434254.120241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434254.120244:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434254.120248:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434254.120257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434254.120275:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434254.120286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434254/real 1789434254] req@ffff9811d1bc6300 x1876350913808640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434254.120297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913808640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434254.120304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434254.120308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434254.120311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434254.120316:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434254.120336:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434254.120346:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434254/real 1789434254] req@ffff9811d1bc5500 x1876350913808768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434254.120357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913808768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434254.120366:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434254.120369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434254.120372:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913808768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434255.014446:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434256.038740:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434257.062102:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434258.086021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434259.112341:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434259.237940:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434259.237950:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434259.237955:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434259.237966:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6628 00000100:00080000:3.0:1789434259.237975:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434259.237988:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434259.238027:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913808896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434259.238060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434259.238064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434259.238065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434259.238070:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6628 00000100:00080000:3.0:1789434259.238074:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434259.238079:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434259.238088:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434259.238102:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913809024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434259.238112:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434259.238116:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434259.238118:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434259.238121:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6628 00000100:00080000:3.0:1789434259.238124:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434259.238129:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434259.238133:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434259.238144:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913809152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434259.238154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434259.238157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434259.238159:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434259.238163:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6628 00000100:00080000:3.0:1789434259.238166:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434259.238170:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434259.238175:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434259.238187:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913809280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434259.238270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913808896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434259.238361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434259.238385:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434259.238410:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434260.134562:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434261.158682:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434262.182413:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789434262.312541:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789434262.312586:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789434262.312784:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434262.313123:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434259/real 1789434262] req@ffff9811d1bc6680 x1876350913808896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434262.313192:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913808896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434262.313203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434262.313207:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434262.313210:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913808896:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434262.313902:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434262.314062:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434259/real 1789434262] req@ffff9811d1bc5c00 x1876350913809024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434262.314079:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913809024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434262.314088:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434262.314092:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434262.314095:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434262.314141:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434262.314215:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434259/real 1789434262] req@ffff9811d1bc5880 x1876350913809152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434262.314233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913809152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434262.314241:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434262.314244:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434262.314247:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434262.314286:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434262.316067:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434259/real 1789434262] req@ffff9811d1bc5f80 x1876350913809280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434262.316095:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913809280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434284 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434262.316106:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434262.316111:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434262.316115:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434263.206718:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434264.231138:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434264.358694:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434264.358713:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434264.358716:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434264.358726:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6633 00000100:00080000:2.0:1789434264.358736:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434264.358749:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434264.358787:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913809408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434264.358814:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434264.358817:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434264.358819:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434264.358824:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6633 00000100:00080000:2.0:1789434264.358827:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434264.358861:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434264.358872:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434264.358884:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913809536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434264.358894:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434264.358897:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434264.358899:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434264.358905:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6633 00000100:00080000:2.0:1789434264.358908:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434264.358912:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434264.358916:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434264.358927:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913809664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434264.358937:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434264.358940:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434264.358941:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434264.358945:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6633 00000100:00080000:2.0:1789434264.358947:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434264.358953:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434264.358958:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434264.358969:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913809792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434264.358981:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913809408:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434264.359059:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434264.359096:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434264/real 1789434264] req@ffff9811d2102d80 x1876350913809408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434264.359115:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913809408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434264.359126:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434264.359130:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434264.359134:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913809408:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434264.359141:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434264.359164:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434264.359176:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434264/real 1789434264] req@ffff9811d2101500 x1876350913809536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434264.359188:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913809536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434264.359195:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434264.359198:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434264.359201:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434264.359206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434264.359223:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434264.359233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434264/real 1789434264] req@ffff9811d2102a00 x1876350913809664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434264.359243:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913809664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434264.359251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434264.359254:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434264.359257:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434264.359281:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809792:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434264.359301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434264.359312:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434264/real 1789434264] req@ffff9811d2103b80 x1876350913809792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434264.359327:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913809792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434264.359335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434264.359338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434264.359341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913809792:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434265.254711:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434266.278599:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434267.302092:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434268.326063:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434269.350550:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434269.478105:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434269.478122:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434269.478128:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434269.478141:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6639 00000100:00080000:3.0:1789434269.478155:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434269.478182:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434269.478235:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913809920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434269.478268:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434269.478272:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434269.478275:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434269.478279:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6639 00000100:00080000:3.0:1789434269.478283:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434269.478289:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434269.478300:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434269.478316:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913810048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434269.478326:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434269.478329:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434269.478331:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434269.478335:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6639 00000100:00080000:3.0:1789434269.478338:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434269.478351:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434269.478357:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434269.478368:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913810176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434269.478378:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434269.478382:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434269.478384:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434269.478398:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6639 00000100:00080000:3.0:1789434269.478402:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434269.478407:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434269.478413:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434269.478424:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913810304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434269.478642:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913809920:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434269.478725:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434269.478781:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434269/real 1789434269] req@ffff9811d1bc5f80 x1876350913809920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434269.478802:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913809920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434269.478815:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434269.478821:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434269.478825:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913809920:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434269.478857:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434269.478881:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434269.478895:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434269/real 1789434269] req@ffff9811d1bc5880 x1876350913810048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434269.478905:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913810048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434269.478913:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434269.478916:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434269.478920:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434269.478926:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434269.478942:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434269.478952:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434269/real 1789434269] req@ffff9811d1bc5c00 x1876350913810176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434269.478963:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913810176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434269.478970:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434269.478973:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434269.478976:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434269.478983:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810304:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434269.479002:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434269.479012:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434269/real 1789434269] req@ffff9811d1bc6680 x1876350913810304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434269.479023:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913810304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434269.479031:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434269.479034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434269.479036:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810304:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434270.374280:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434271.399448:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434272.422087:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434273.446680:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434274.470801:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434274.601431:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434274.601445:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434274.601451:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434274.601463:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6644 00000100:00080000:3.0:1789434274.601476:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434274.601491:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434274.601541:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913810432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434274.601582:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434274.601586:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434274.601589:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434274.601594:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6644 00000100:00080000:3.0:1789434274.601597:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434274.601602:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434274.601615:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434274.601627:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913810560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434274.601637:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434274.601640:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434274.601642:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434274.601645:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6644 00000100:00080000:3.0:1789434274.601648:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434274.601652:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434274.601656:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434274.601664:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913810688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434274.601680:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434274.601683:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434274.601685:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434274.601688:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6644 00000100:00080000:3.0:1789434274.601691:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434274.601694:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434274.601698:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434274.601707:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913810816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434274.601883:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913810432:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434274.601962:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434274.602011:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434274/real 1789434274] req@ffff9811d1bc4000 x1876350913810432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434274.602030:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913810432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434274.602041:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434274.602045:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434274.602048:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913810432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434274.602054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434274.602075:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434274.602086:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434274/real 1789434274] req@ffff9811d1bc7800 x1876350913810560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434274.602098:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913810560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434274.602106:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434274.602109:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434274.602111:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434274.602116:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434274.602134:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434274.602145:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434274/real 1789434274] req@ffff9811d1bc5180 x1876350913810688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434274.602156:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913810688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434274.602163:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434274.602166:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434274.602169:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434274.602174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810816:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434274.602195:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434274.602205:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434274/real 1789434274] req@ffff9811d1bc6a00 x1876350913810816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434274.602215:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913810816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434274.602230:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434274.602233:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434274.602237:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913810816:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434275.497137:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434276.520492:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434277.543773:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434278.566049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434279.590067:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434279.717937:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434279.717948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434279.717952:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434279.717963:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6649 00000100:00080000:3.0:1789434279.717971:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434279.717981:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434279.718016:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913810944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434279.718049:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434279.718053:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434279.718055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434279.718059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6649 00000100:00080000:3.0:1789434279.718062:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434279.718067:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434279.718077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434279.718089:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913811072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434279.718099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434279.718102:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434279.718103:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434279.718107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6649 00000100:00080000:3.0:1789434279.718110:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434279.718114:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434279.718117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434279.718126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913811200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434279.718135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434279.718138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434279.718139:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434279.718143:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6649 00000100:00080000:3.0:1789434279.718146:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434279.718150:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434279.718154:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434279.718163:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913811328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434279.718248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913810944:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434279.718302:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434279.718344:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434279/real 1789434279] req@ffff9811d1bc7480 x1876350913810944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434279.718372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913810944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434279.718383:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434279.718387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434279.718391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913810944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434279.718397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434279.718419:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434279.718430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434279/real 1789434279] req@ffff9811d1bc6a00 x1876350913811072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434279.718441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913811072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434279.718449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434279.718453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434279.718456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434279.718461:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434279.718477:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434279.718486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434279/real 1789434279] req@ffff9811d1bc5180 x1876350913811200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434279.718497:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913811200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434279.718505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434279.718508:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434279.718511:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434279.718517:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811328:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434279.718533:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434279.718542:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434279/real 1789434279] req@ffff9811d1bc7800 x1876350913811328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434279.718552:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913811328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434279.718571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434279.718574:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434279.718577:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811328:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434280.614386:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434281.638055:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434282.662362:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434283.686800:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434284.710666:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434284.837966:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434284.837978:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434284.837983:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434284.837994:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6654 00000100:00080000:1.0:1789434284.838003:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434284.838013:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434284.838051:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913811456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434284.838079:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434284.838082:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434284.838084:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434284.838088:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6654 00000100:00080000:1.0:1789434284.838091:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434284.838096:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434284.838105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434284.838117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913811584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434284.838127:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434284.838130:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434284.838132:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434284.838137:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913811456:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434284.838138:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6654 00000100:00080000:1.0:1789434284.838143:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434284.838148:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434284.838152:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434284.838162:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913811712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434284.838190:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434284.838193:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434284.838195:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434284.838198:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6654 00000800:00000100:0.0:1789434284.838203:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434284.838205:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434284.838212:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434284.838217:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434284.838227:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913811840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434284.838249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434284/real 1789434284] req@ffff9811d0bde680 x1876350913811456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434284.838267:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913811456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434284.838277:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434284.838280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434284.838284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913811456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434284.838305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434284.838342:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434284.838355:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434284/real 1789434284] req@ffff9811d0bdf480 x1876350913811584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434284.838368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913811584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434284.838375:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434284.838378:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434284.838381:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434284.838387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434284.838406:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434284.838417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434284/real 1789434284] req@ffff9811d0bdfb80 x1876350913811712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434284.838455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913811712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434284.838463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434284.838466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434284.838469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434284.838475:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811840:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434284.838495:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434284.838506:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434284/real 1789434284] req@ffff9811d0bded80 x1876350913811840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434284.838517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913811840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434284.838525:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434284.838528:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434284.838531:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913811840:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434285.734362:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434286.758024:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434287.782649:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434288.806348:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434289.830477:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434289.957939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434289.957952:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434289.957957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434289.957975:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6659 00000100:00080000:2.0:1789434289.957983:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434289.957995:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434289.958029:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913811968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434289.958059:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434289.958061:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434289.958063:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434289.958066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6659 00000100:00080000:2.0:1789434289.958069:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434289.958072:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434289.958080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434289.958089:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913812096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434289.958095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434289.958097:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434289.958098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434289.958100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6659 00000100:00080000:2.0:1789434289.958101:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434289.958103:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434289.958105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434289.958110:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913812224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434289.958117:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434289.958119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434289.958120:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434289.958122:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6659 00000100:00080000:2.0:1789434289.958124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434289.958125:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434289.958128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434289.958133:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913812352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434289.958333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913811968:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434289.958394:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434289.958441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434289/real 1789434289] req@ffff9811d2101c00 x1876350913811968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434289.958459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913811968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434289.958479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434289.958483:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434289.958487:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913811968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434289.958493:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434289.958514:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434289.958526:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434289/real 1789434289] req@ffff9811d2101880 x1876350913812096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434289.958538:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913812096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434289.958545:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434289.958548:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434289.958551:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434289.958557:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434289.958574:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434289.958585:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434289/real 1789434289] req@ffff9811d2101180 x1876350913812224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434289.958596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913812224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434289.958604:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434289.958608:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434289.958611:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434289.958617:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812352:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434289.958633:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434289.958643:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434289/real 1789434289] req@ffff9811d2101f80 x1876350913812352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434289.958653:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913812352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434289.958660:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434289.958664:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434289.958666:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812352:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434290.854029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434291.878321:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434292.902106:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434293.926086:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434294.951426:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434295.078045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434295.078057:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434295.078061:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434295.078075:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6664 00000100:00080000:3.0:1789434295.078086:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434295.078100:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434295.078142:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913812480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434295.078174:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434295.078178:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434295.078180:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434295.078184:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6664 00000100:00080000:3.0:1789434295.078188:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434295.078192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434295.078202:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434295.078215:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913812608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434295.078223:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913812480:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434295.078229:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434295.078233:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434295.078235:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434295.078239:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6664 00000100:00080000:3.0:1789434295.078243:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434295.078260:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434295.078265:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434295.078277:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913812736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434295.078288:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434295.078291:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434295.078293:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434295.078296:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6664 00000800:00000100:0.0:1789434295.078301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434295.078303:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434295.078310:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434295.078315:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434295.078328:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913812864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434295.078339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434295/real 1789434295] req@ffff9811d1bc4000 x1876350913812480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434295.078354:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913812480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434295.078363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434295.078366:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434295.078369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913812480:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434295.078389:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434295.078406:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434295.078413:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434295/real 1789434295] req@ffff9811d1bc6680 x1876350913812608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434295.078420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913812608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434295.078427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434295.078430:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434295.078433:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434295.078438:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434295.078456:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434295.078466:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434295/real 1789434295] req@ffff9811d1bc5c00 x1876350913812736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434295.078491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913812736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434295.078503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434295.078506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434295.078509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434295.078514:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812864:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434295.078533:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434295.078543:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434295/real 1789434295] req@ffff9811d1bc5880 x1876350913812864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434295.078561:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913812864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434295.078569:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434295.078572:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434295.078575:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913812864:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434295.973971:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434296.998101:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434298.022148:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434299.046060:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434300.070705:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434300.197920:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434300.197927:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434300.197931:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434300.197940:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6669 00000100:00080000:3.0:1789434300.197946:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434300.197954:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434300.197982:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913812992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434300.198010:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434300.198013:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434300.198014:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434300.198019:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6669 00000100:00080000:3.0:1789434300.198022:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434300.198028:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434300.198038:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434300.198048:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913812992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434300.198052:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913813120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434300.198062:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434300.198064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434300.198066:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434300.198068:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6669 00000100:00080000:3.0:1789434300.198071:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434300.198074:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434300.198079:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434300.198089:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913813248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434300.198099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434300.198103:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434300.198104:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434300.198108:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6669 00000100:00080000:3.0:1789434300.198112:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789434300.198114:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434300.198119:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434300.198126:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434300.198140:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913813376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434300.198151:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434300/real 1789434300] req@ffff9811d1bc5f80 x1876350913812992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434300.198169:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913812992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434300.198183:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434300.198186:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434300.198188:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913812992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434300.198205:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434300.198234:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434300.198243:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434300/real 1789434300] req@ffff9811d1bc5500 x1876350913813120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434300.198250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913813120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434300.198259:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434300.198261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434300.198263:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434300.198267:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434300.198276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434300.198281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434300/real 1789434300] req@ffff9811d1bc6300 x1876350913813248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434300.198289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913813248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434300.198297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434300.198300:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434300.198303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434300.198309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813376:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434300.198327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434300.198336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434300/real 1789434300] req@ffff9811d1bc4700 x1876350913813376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434300.198343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913813376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434300.198348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434300.198350:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434300.198352:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813376:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434301.094021:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434302.118037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434303.142903:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434304.167750:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434305.189987:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434305.317912:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434305.317933:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434305.317938:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434305.317948:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6674 00000100:00080000:3.0:1789434305.317956:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434305.317965:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434305.317995:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913813504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434305.318017:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434305.318021:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434305.318023:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434305.318027:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6674 00000100:00080000:3.0:1789434305.318030:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434305.318035:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434305.318044:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434305.318058:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913813632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434305.318065:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913813504:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434305.318083:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434305.318086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434305.318088:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434305.318093:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6674 00000100:00080000:3.0:1789434305.318097:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434305.318102:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434305.318107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434305.318120:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913813760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434305.318123:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434305.318135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434305.318138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434305.318140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434305.318145:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6674 00000100:00080000:3.0:1789434305.318149:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434305.318155:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434305.318160:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434305.318169:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434305/real 1789434305] req@ffff9811d1bc7100 x1876350913813504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434305.318186:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913813888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434305.318206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913813504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434305.318217:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434305.318221:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434305.318224:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913813504:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434305.318247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434305.318268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434305.318281:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434305/real 1789434305] req@ffff9811d1bc4700 x1876350913813632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434305.318305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913813632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434305.318313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434305.318316:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434305.318319:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434305.318324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434305.318344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434305.318354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434305/real 1789434305] req@ffff9811d1bc6300 x1876350913813760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434305.318365:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913813760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434305.318372:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434305.318376:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434305.318378:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434305.318382:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813888:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434305.318398:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434305.318404:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434305/real 1789434305] req@ffff9811d1bc5500 x1876350913813888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434305.318411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913813888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434305.318418:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434305.318421:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434305.318424:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913813888:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434306.214206:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434307.238264:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434308.262179:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434309.286262:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434310.310019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434310.438084:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434310.438096:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434310.438101:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434310.438112:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6680 00000100:00080000:3.0:1789434310.438120:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434310.438131:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434310.438186:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913814016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434310.438220:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434310.438224:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434310.438227:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434310.438232:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6680 00000100:00080000:3.0:1789434310.438235:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434310.438241:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434310.438251:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434310.438265:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913814144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434310.438276:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913814016:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434310.438278:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434310.438285:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434310.438287:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434310.438291:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6680 00000100:00080000:3.0:1789434310.438294:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434310.438299:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434310.438304:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434310.438316:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913814272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434310.438327:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434310.438330:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434310.438332:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434310.438336:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6680 00000100:00080000:3.0:1789434310.438339:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434310.438347:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434310.438352:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789434310.438358:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434310.438365:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913814400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434310.438402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434310/real 1789434310] req@ffff9811d1bc5f80 x1876350913814016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434310.438420:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913814016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434310.438430:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434310.438434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434310.438438:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913814016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434310.438459:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434310.438479:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434310.438491:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434310/real 1789434310] req@ffff9811d1bc5880 x1876350913814144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434310.438503:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913814144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434310.438511:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434310.438514:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434310.438517:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434310.438522:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434310.438540:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434310.438550:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434310/real 1789434310] req@ffff9811d1bc5c00 x1876350913814272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434310.438561:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913814272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434310.438573:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434310.438576:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434310.438579:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434310.438585:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814400:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434310.438603:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434310.438613:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434310/real 1789434310] req@ffff9811d1bc6680 x1876350913814400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434310.438624:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913814400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434310.438632:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434310.438635:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434310.438638:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814400:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434311.334062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434312.358175:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434313.382074:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434314.406020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434315.430132:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434315.558062:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434315.558073:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434315.558077:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434315.558088:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6685 00000100:00080000:3.0:1789434315.558109:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434315.558123:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434315.558178:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913814528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434315.558339:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913814528:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434315.558431:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434315.558467:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434315.558471:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434315.558473:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434315.558477:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6685 00000100:00080000:3.0:1789434315.558480:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434315.558484:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434315.558500:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434315.558511:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913814656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434315.558568:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434315/real 1789434315] req@ffff9811d1bc4000 x1876350913814528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434315.558597:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434315.558600:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913814528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434315.558617:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434315.558627:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434315.558633:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434315.558638:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6685 00000100:00080000:0.0:1789434315.558645:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:3.0:1789434315.558650:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789434315.558655:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913814528:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434315.558664:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434315.558676:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434315.558687:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434315.558712:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913814784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434315.558725:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434315.558742:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434315.558749:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434315/real 1789434315] req@ffff9811d1bc7800 x1876350913814656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434315.558750:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434315.558755:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434315.558762:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6685 00000100:00080000:3.0:1789434315.558768:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434315.558778:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434315.558785:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434315.558788:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913814656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434315.558801:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913814912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434315.558803:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434315.558820:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434315.558827:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434315.558878:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434315.558905:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434315.558916:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434315/real 1789434315] req@ffff9811d1bc5180 x1876350913814784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434315.558927:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913814784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434315.558935:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434315.558938:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434315.558941:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434315.558947:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814912:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434315.558964:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434315.558974:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434315/real 1789434315] req@ffff9811d1bc6a00 x1876350913814912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434315.558984:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913814912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434315.558992:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434315.558994:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434315.558997:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913814912:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434316.454045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434317.478031:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434318.502013:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434319.526025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434320.550620:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434320.677931:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434320.677941:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434320.677946:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434320.677956:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6690 00000100:00080000:3.0:1789434320.677965:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434320.677976:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434320.678020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913815040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434320.678115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434320.678119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434320.678121:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434320.678127:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913815040:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434320.678127:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6690 00000100:00080000:3.0:1789434320.678133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434320.678140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434320.678150:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434320.678176:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913815168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434320.678189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789434320.678193:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434320.678195:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434320.678198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434320.678203:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6690 00000100:00080000:3.0:1789434320.678208:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434320.678219:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434320.678224:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434320.678235:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434320/real 1789434320] req@ffff9811d1bc7480 x1876350913815040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434320.678239:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913815296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434320.678257:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434320.678259:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434320.678261:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434320.678264:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6690 00000100:00080000:3.0:1789434320.678267:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434320.678271:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434320.678275:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434320.678284:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913815424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434320.678454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913815040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434320.678465:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434320.678469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434320.678473:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913815040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434320.678495:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434320.678519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434320.678531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434320/real 1789434320] req@ffff9811d1bc6a00 x1876350913815168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434320.678549:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913815168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434320.678558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434320.678561:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434320.678564:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434320.678569:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434320.678585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434320.678594:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434320/real 1789434320] req@ffff9811d1bc5180 x1876350913815296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434320.678605:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913815296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434320.678613:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434320.678616:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434320.678619:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434320.678625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815424:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434320.678644:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434320.678654:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434320/real 1789434320] req@ffff9811d1bc7800 x1876350913815424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434320.678663:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913815424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434320.678676:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434320.678679:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434320.678682:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815424:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434321.574726:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434322.598255:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434323.622076:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434324.648025:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434325.670365:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434325.797929:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434325.797936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434325.797939:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434325.797948:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6695 00000100:00080000:3.0:1789434325.797957:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434325.797966:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434325.798002:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913815552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434325.798036:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434325.798040:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434325.798042:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434325.798046:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6695 00000100:00080000:3.0:1789434325.798050:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434325.798056:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434325.798066:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434325.798073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913815552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434325.798081:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913815680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434325.798093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434325.798096:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434325.798098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434325.798102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6695 00000100:00080000:3.0:1789434325.798106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434325.798110:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434325.798116:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434325.798126:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913815808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434325.798135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434325.798138:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434325.798140:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434325.798144:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6695 00000100:00080000:3.0:1789434325.798147:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434325.798153:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434325.798158:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434325.798169:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913815936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434325.798173:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434325.798210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434325.798235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815936:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434326.694030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434327.720033:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434328.742118:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434328.870203:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434328.870242:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434328.870256:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434328.870302:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434328.870311:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434328.870319:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434328.872493:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434325/real 1789434328] req@ffff9811d1bc5880 x1876350913815936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434328.872526:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913815936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434328.872538:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434328.872542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434328.872546:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815936:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789434328.872554:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434325/real 1789434328] req@ffff9811d1bc6680 x1876350913815680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434328.872565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913815680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434328.872572:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434328.872575:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434328.872578:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434328.872584:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434325/real 1789434328] req@ffff9811d1bc5c00 x1876350913815808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434328.872593:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913815808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434328.872599:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434328.872602:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434328.872605:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913815808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434328.872612:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434325/real 1789434328] req@ffff9811d1bc4000 x1876350913815552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434328.872623:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913815552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434328.872631:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434328.872634:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434328.872636:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913815552:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789434329.766506:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434330.790036:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434330.918055:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434330.918225:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434330.918230:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434330.918239:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6700 00000100:00080000:3.0:1789434330.918247:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434330.918258:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434330.918288:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913816064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434330.918473:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434330.918477:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434330.918479:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434330.918484:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6700 00000100:00080000:3.0:1789434330.918489:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434330.918496:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434330.918505:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434330.918516:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913816064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434330.918523:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913816192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434330.918534:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434330.918538:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434330.918540:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434330.918543:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6700 00000100:00080000:3.0:1789434330.918548:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434330.918553:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434330.918558:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434330.918570:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913816320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434330.918579:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434330.918582:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434330.918587:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434330.918589:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434330.918594:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6700 00000100:00080000:3.0:1789434330.918598:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434330.918603:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434330.918609:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434330.918616:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434330/real 1789434330] req@ffff9811d1bc5f80 x1876350913816064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434330.918625:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913816448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434330.918638:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913816064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434330.918649:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434330.918654:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434330.918658:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913816064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434330.918680:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434330.918700:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434330.918712:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434330/real 1789434330] req@ffff9811d1bc5500 x1876350913816192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434330.918723:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913816192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434330.918731:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434330.918734:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434330.918737:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434330.918743:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434330.918761:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434330.918771:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434330/real 1789434330] req@ffff9811d1bc6300 x1876350913816320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434330.918783:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913816320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434330.918796:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434330.918799:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434330.918802:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434330.918813:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816448:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434330.918877:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434330.918888:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434330/real 1789434330] req@ffff9811d1bc4700 x1876350913816448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434330.918900:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913816448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434330.918908:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434330.918911:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434330.918920:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816448:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434331.814227:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434332.838202:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434333.862181:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434334.886118:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434335.910021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434336.037954:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434336.037964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434336.037969:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434336.037978:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6705 00000100:00080000:3.0:1789434336.037987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434336.037997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434336.038032:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913816576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434336.038065:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434336.038069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434336.038070:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434336.038075:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6705 00000100:00080000:3.0:1789434336.038079:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434336.038083:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434336.038093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434336.038109:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913816704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434336.038119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434336.038123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913816576:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434336.038124:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434336.038126:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434336.038130:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6705 00000100:00080000:3.0:1789434336.038133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434336.038140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434336.038145:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434336.038157:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913816832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434336.038166:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434336.038169:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434336.038171:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434336.038174:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6705 00000100:00080000:3.0:1789434336.038178:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434336.038182:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434336.038187:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434336.038197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913816960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434336.038202:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434336.038249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434336/real 1789434336] req@ffff9811d1bc7100 x1876350913816576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434336.038266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913816576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434336.038276:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434336.038280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434336.038284:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913816576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434336.038305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434336.038326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434336.038339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434336/real 1789434336] req@ffff9811d1bc4700 x1876350913816704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434336.038350:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913816704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434336.038359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434336.038362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434336.038365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434336.038371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434336.038389:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434336.038401:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434336/real 1789434336] req@ffff9811d1bc6300 x1876350913816832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434336.038411:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913816832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434336.038419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434336.038422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434336.038425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434336.038431:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816960:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434336.038451:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434336.038461:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434336/real 1789434336] req@ffff9811d1bc5500 x1876350913816960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434336.038473:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913816960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434336.038481:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434336.038489:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434336.038492:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913816960:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434336.934177:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434337.958072:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434338.982413:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434340.006111:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434341.030439:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434341.157953:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434341.157963:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434341.157967:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434341.157976:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6710 00000100:00080000:3.0:1789434341.157985:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434341.157997:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434341.158033:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913817088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434341.158072:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434341.158076:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434341.158078:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434341.158082:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6710 00000100:00080000:3.0:1789434341.158085:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434341.158090:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434341.158099:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434341.158112:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913817216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434341.158122:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434341.158126:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434341.158128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434341.158131:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6710 00000100:00080000:3.0:1789434341.158134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434341.158139:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434341.158144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434341.158155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913817344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434341.158163:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913817088:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434341.158169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434341.158174:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434341.158176:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434341.158180:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6710 00000100:00080000:3.0:1789434341.158184:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434341.158189:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434341.158194:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434341.158207:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913817472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434341.158227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434341.158265:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434341/real 1789434341] req@ffff9811d1bc5f80 x1876350913817088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434341.158282:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913817088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434341.158292:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434341.158296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434341.158299:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913817088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434341.158307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434341.158328:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434341.158340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434341/real 1789434341] req@ffff9811d1bc4000 x1876350913817216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434341.158351:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913817216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434341.158360:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434341.158362:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434341.158365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434341.158387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434341.158405:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434341.158415:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434341/real 1789434341] req@ffff9811d1bc5c00 x1876350913817344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434341.158426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913817344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434341.158433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434341.158436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434341.158439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434341.158445:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817472:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434341.158462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434341.158471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434341/real 1789434341] req@ffff9811d1bc6680 x1876350913817472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434341.158481:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913817472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434341.158493:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434341.158496:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434341.158500:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817472:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434342.054050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434343.078464:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434344.102064:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434345.126027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434346.150619:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434346.278098:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434346.278107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434346.278112:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434346.278122:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6715 00000100:00080000:3.0:1789434346.278131:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434346.278143:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434346.278179:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913817600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434346.278212:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434346.278216:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434346.278217:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434346.278222:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6715 00000100:00080000:3.0:1789434346.278225:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434346.278230:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434346.278240:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434346.278518:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913817600:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434346.278581:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434346.278620:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434346/real 1789434346] req@ffff9811d1bc5880 x1876350913817600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434346.278636:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913817600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434346.278647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434346.278650:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434346.278654:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913817600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434346.278814:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913817728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434346.278827:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434346.278830:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434346.278853:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434346.278857:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6715 00000100:00080000:3.0:1789434346.278861:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434346.278867:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434346.278883:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434346.278897:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913817856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434346.278907:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434346.278910:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434346.278912:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434346.278916:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6715 00000100:00080000:3.0:1789434346.278919:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434346.278923:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434346.278927:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434346.278939:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913817984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434346.279019:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434346.279046:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434346.279060:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434346/real 1789434346] req@ffff9811d1bc7800 x1876350913817728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434346.279072:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913817728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434346.279080:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434346.279084:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434346.279087:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434346.279093:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434346.279113:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434346.279123:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434346/real 1789434346] req@ffff9811d1bc5180 x1876350913817856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434346.279132:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913817856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434346.279138:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434346.279141:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434346.279143:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434346.279149:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817984:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434346.279167:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434346.279177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434346/real 1789434346] req@ffff9811d1bc6a00 x1876350913817984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434346.279187:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913817984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434346.279195:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434346.279198:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434346.279200:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913817984:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434347.174017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434348.198081:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434349.222034:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434350.246076:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434351.270000:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434351.397967:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434351.397978:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434351.397982:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434351.397992:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6720 00000100:00080000:0.0:1789434351.398000:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434351.398012:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434351.398046:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913818112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434351.398071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434351.398074:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434351.398075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434351.398078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6720 00000100:00080000:0.0:1789434351.398081:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434351.398084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434351.398091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434351.398104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913818240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434351.398115:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789434351.398118:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913818112:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434351.398119:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434351.398123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434351.398129:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6720 00000100:00080000:0.0:1789434351.398133:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434351.398138:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434351.398143:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434351.398156:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913818368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434351.398163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434351.398165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434351.398166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434351.398169:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6720 00000100:00080000:0.0:1789434351.398172:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434351.398175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434351.398179:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434351.398186:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913818496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434351.398192:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434351.398238:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434351/real 1789434351] req@ffff9811c8cd5500 x1876350913818112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434351.398255:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913818112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434351.398265:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434351.398269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434351.398273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913818112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434351.398294:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434351.398316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434351.398328:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434351/real 1789434351] req@ffff9811c8cd4a80 x1876350913818240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434351.398340:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913818240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434351.398348:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434351.398351:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434351.398354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434351.398360:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434351.398377:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434351.398387:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434351/real 1789434351] req@ffff9811c8cd7b80 x1876350913818368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434351.398398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913818368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434351.398406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434351.398409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434351.398412:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434351.398418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818496:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434351.398437:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434351.398447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434351/real 1789434351] req@ffff9811c8cd6300 x1876350913818496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434351.398459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913818496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434351.398466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434351.398479:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434351.398486:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434352.294084:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434353.318042:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434354.342313:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434355.366026:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434356.390172:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434356.518189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434356.518204:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434356.518208:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434356.518221:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6726 00000100:00080000:0.0:1789434356.518233:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434356.518248:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434356.518299:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913818624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434356.518337:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434356.518341:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434356.518343:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434356.518350:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6726 00000100:00080000:0.0:1789434356.518354:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434356.518361:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434356.518374:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434356.518387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913818624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434356.518392:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913818752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434356.518409:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434356.518412:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434356.518415:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434356.518420:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6726 00000100:00080000:0.0:1789434356.518425:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434356.518433:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434356.518438:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434356.518451:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913818880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434356.518464:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434356.518466:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434356.518472:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434356.518474:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434356.518481:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6726 00000100:00080000:0.0:1789434356.518486:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434356.518495:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434356.518501:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434356.518515:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913819008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434356.518522:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434356/real 1789434356] req@ffff9811c8cd4000 x1876350913818624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434356.518565:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913818624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434356.518578:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434356.518584:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434356.518587:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913818624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434356.518612:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434356.518637:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434356.518651:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434356/real 1789434356] req@ffff9811c8cd6a00 x1876350913818752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434356.518663:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913818752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434356.518672:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434356.518675:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434356.518679:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434356.518685:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434356.518707:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434356.518718:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434356/real 1789434356] req@ffff9811c8cd5180 x1876350913818880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434356.518729:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913818880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434356.518737:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434356.518740:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434356.518742:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913818880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434356.518749:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819008:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434356.518768:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434356.518778:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434356/real 1789434356] req@ffff9811c8cd7800 x1876350913819008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434356.518947:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913819008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434356.518956:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434356.518959:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434356.518962:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434357.414346:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434358.438069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434359.462164:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434360.486549:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434361.510451:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434361.638049:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434361.638065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434361.638070:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434361.638080:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6731 00000100:00080000:3.0:1789434361.638089:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434361.638101:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434361.638137:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913819136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434361.638205:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913819136:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434361.638261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434361.638291:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434361.638295:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434361.638296:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434361.638300:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6731 00000100:00080000:3.0:1789434361.638304:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434361.638309:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434361.638319:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434361.638344:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913819264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434361.638428:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434361/real 1789434361] req@ffff9811d1bc7480 x1876350913819136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434361.638441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913819136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434361.638447:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434361.638450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434361.638452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913819136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434361.638469:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434361.638485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434361.638493:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434361/real 1789434361] req@ffff9811d1bc6a00 x1876350913819264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434361.638500:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913819264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434361.638504:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434361.638506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434361.638508:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819264:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434361.639078:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434361.639082:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434361.639083:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434361.639088:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6731 00000100:00080000:3.0:1789434361.639092:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434361.639098:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434361.639103:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434361.639117:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913819392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434361.639232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434361.639258:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434361.639280:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434361.639284:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434361.639286:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434361.639290:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6731 00000100:00080000:3.0:1789434361.639294:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434361.639298:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434361.639303:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434361.639315:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913819520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434361.639407:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434361/real 1789434361] req@ffff9811d1bc5180 x1876350913819392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434361.639419:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913819392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434361.639427:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434361.639430:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434361.639433:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434361.639445:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819520:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434361.639466:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434361.639477:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434361/real 1789434361] req@ffff9811d1bc7800 x1876350913819520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434361.639488:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913819520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434361.639496:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434361.639499:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434361.639501:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434362.534602:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434363.558161:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434364.582777:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434365.608062:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434366.630020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434366.757937:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434366.757949:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434366.757953:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434366.757963:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6736 00000100:00080000:0.0:1789434366.757971:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434366.757982:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434366.758016:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913819648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434366.758044:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434366.758048:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434366.758049:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434366.758054:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6736 00000100:00080000:0.0:1789434366.758057:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434366.758060:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434366.758068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434366.758078:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913819776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434366.758086:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913819648:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434366.758087:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434366.758091:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434366.758094:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434366.758098:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6736 00000100:00080000:0.0:1789434366.758100:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434366.758102:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434366.758105:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434366.758112:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913819904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434366.758119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434366.758121:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434366.758122:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434366.758124:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6736 00000100:00080000:0.0:1789434366.758126:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434366.758128:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434366.758131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434366.758137:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913820032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434366.758154:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434366.758192:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434366/real 1789434366] req@ffff9811c8cd7480 x1876350913819648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434366.758210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913819648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434366.758221:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434366.758225:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434366.758229:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913819648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434366.758251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434366.758273:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434366.758287:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434366/real 1789434366] req@ffff9811c8cd7100 x1876350913819776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434366.758299:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913819776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434366.758313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434366.758317:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434366.758321:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434366.758326:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434366.758345:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434366.758356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434366/real 1789434366] req@ffff9811c8cd5c00 x1876350913819904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434366.758368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913819904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434366.758376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434366.758379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434366.758382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913819904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434366.758388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820032:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434366.758407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434366.758418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434366/real 1789434366] req@ffff9811c8cd4700 x1876350913820032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434366.758429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913820032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434366.758437:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434366.758440:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434366.758444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434367.654208:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434368.678018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434369.702743:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434370.726003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434371.750017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434371.878114:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434371.878121:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434371.878124:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434371.878132:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6741 00000100:00080000:3.0:1789434371.878139:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434371.878149:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434371.878176:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913820160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434371.878231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913820160:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434371.878251:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434371.878254:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434371.878256:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434371.878260:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6741 00000100:00080000:3.0:1789434371.878263:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434371.878268:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434371.878277:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434371.878291:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913820288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434371.878298:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434371.878300:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434371.878303:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434371.878305:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434371.878308:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6741 00000100:00080000:3.0:1789434371.878312:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434371.878317:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434371.878321:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434371.878329:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913820416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434371.878330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434371/real 1789434371] req@ffff9811d1bc5880 x1876350913820160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434371.878340:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434371.878343:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434371.878345:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434371.878346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913820160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434371.878348:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6741 00000100:00080000:3.0:1789434371.878350:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434371.878354:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00080000:0.0:1789434371.878356:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789434371.878358:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789434371.878359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434371.878363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913820160:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434371.878366:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913820544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434371.878387:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434371.878408:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434371.878419:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434371/real 1789434371] req@ffff9811d1bc6680 x1876350913820288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434371.878427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913820288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434371.878432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434371.878434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434371.878436:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434371.878439:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434371.878451:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434371.878756:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434371/real 1789434371] req@ffff9811d1bc5c00 x1876350913820416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434371.878771:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913820416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434371.878778:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434371.878782:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434371.878785:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434371.878791:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820544:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434371.878813:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434371.878824:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434371/real 1789434371] req@ffff9811d1bc4000 x1876350913820544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434371.878856:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913820544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434371.878861:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434371.878863:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434371.878865:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434372.773986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434373.798860:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434374.822643:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434375.846113:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434376.871717:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434376.997954:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434376.997973:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434376.997977:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434376.997987:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6746 00000100:00080000:3.0:1789434376.997995:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434376.998006:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434376.998040:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913820672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434376.998065:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434376.998069:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434376.998072:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434376.998076:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6746 00000100:00080000:3.0:1789434376.998079:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434376.998083:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434376.998093:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434376.998110:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913820800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434376.998112:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913820672:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434376.998129:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434376.998131:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434376.998132:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434376.998135:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6746 00000100:00080000:3.0:1789434376.998138:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434376.998141:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434376.998145:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434376.998154:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913820928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434376.998162:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434376.998164:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434376.998165:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434376.998167:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6746 00000800:00000100:0.0:1789434376.998173:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434376.998181:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434376.998188:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434376.998193:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434376.998206:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913821056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434376.998214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434376/real 1789434376] req@ffff9811d1bc5f80 x1876350913820672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434376.998234:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913820672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434376.998256:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434376.998261:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434376.998264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913820672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434376.998341:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434376.998367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434376.998379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434376/real 1789434376] req@ffff9811d1bc5500 x1876350913820800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434376.998397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913820800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434376.998406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434376.998409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434376.998412:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434376.998418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434376.998436:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434376.998447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434376/real 1789434376] req@ffff9811d1bc6300 x1876350913820928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434376.998457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913820928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434376.998464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434376.998468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434376.998470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913820928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434376.998477:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434376.998495:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434376.998505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434376/real 1789434376] req@ffff9811d1bc4700 x1876350913821056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434376.998515:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913821056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434376.998528:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434376.998531:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434376.998534:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434377.894272:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434378.918048:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434379.942198:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434380.967726:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434381.990616:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434382.117997:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434382.118007:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434382.118011:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434382.118021:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6751 00000100:00080000:2.0:1789434382.118029:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434382.118040:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434382.118076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913821184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434382.118103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434382.118107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434382.118108:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434382.118112:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6751 00000100:00080000:2.0:1789434382.118115:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434382.118119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434382.118128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434382.118146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913821312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434382.118155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434382.118158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434382.118159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434382.118162:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6751 00000100:00080000:2.0:1789434382.118165:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434382.118169:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434382.118173:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434382.118181:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913821440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434382.118189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434382.118192:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434382.118193:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434382.118196:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6751 00000100:00080000:2.0:1789434382.118199:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434382.118203:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434382.118207:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434382.118217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913821568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434382.118284:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913821184:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434382.118352:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434382.118385:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434382/real 1789434382] req@ffff9811d2100e00 x1876350913821184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434382.118406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913821184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434382.118416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434382.118420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434382.118423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913821184:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434382.118430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434382.118448:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434382.118458:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434382/real 1789434382] req@ffff9811d2101f80 x1876350913821312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434382.118479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913821312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434382.118487:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434382.118490:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434382.118494:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434382.118499:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434382.118517:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434382.118526:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434382/real 1789434382] req@ffff9811d2101180 x1876350913821440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434382.118537:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913821440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434382.118544:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434382.118547:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434382.118550:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434382.118555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821568:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434382.118736:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434382.118747:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434382/real 1789434382] req@ffff9811d2101880 x1876350913821568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434382.118758:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913821568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434382.118766:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434382.118769:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434382.118772:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434383.013968:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434384.039296:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434385.062022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434386.086028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434387.110807:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434387.237942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434387.237952:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434387.237956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434387.237966:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6756 00000100:00080000:0.0:1789434387.237975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434387.237986:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434387.238019:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913821696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434387.238050:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434387.238054:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434387.238055:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434387.238059:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6756 00000100:00080000:0.0:1789434387.238063:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434387.238068:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434387.238077:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434387.238091:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913821824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434387.238101:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434387.238104:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434387.238106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434387.238109:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6756 00000100:00080000:0.0:1789434387.238112:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434387.238115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434387.238119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434387.238129:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913821952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434387.238150:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434387.238153:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434387.238162:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434387.238166:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6756 00000100:00080000:0.0:1789434387.238169:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434387.238174:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434387.238179:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434387.238189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913822080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434387.238324:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913821696:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434387.238387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434387.238423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434387/real 1789434387] req@ffff9811c8cd6680 x1876350913821696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434387.238447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913821696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434387.238456:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434387.238460:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434387.238463:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913821696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434387.238470:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434387.238492:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434387.238505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434387/real 1789434387] req@ffff9811c8cd4700 x1876350913821824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434387.238516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913821824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434387.238523:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434387.238526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434387.238529:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434387.238535:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434387.238551:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434387.238561:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434387/real 1789434387] req@ffff9811c8cd5c00 x1876350913821952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434387.238586:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913821952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434387.238599:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434387.238602:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434387.238605:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913821952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434387.238611:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434387.238631:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434387.238641:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434387/real 1789434387] req@ffff9811c8cd7100 x1876350913822080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434387.238652:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913822080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434387.238660:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434387.238663:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434387.238666:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434388.134007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434389.158005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434390.181995:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434391.206171:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434392.230029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434392.358143:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434392.358153:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434392.358157:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434392.358168:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6761 00000100:00080000:1.0:1789434392.358177:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434392.358188:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434392.358222:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913822208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434392.358255:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434392.358259:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434392.358261:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434392.358265:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6761 00000100:00080000:1.0:1789434392.358268:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434392.358272:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434392.358281:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434392.358291:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913822336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434392.358299:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434392.358302:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434392.358304:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434392.358307:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6761 00000100:00080000:1.0:1789434392.358310:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434392.358316:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434392.358320:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434392.358326:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913822208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434392.358331:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913822464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434392.358343:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434392.358346:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434392.358347:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434392.358350:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6761 00000100:00080000:1.0:1789434392.358354:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434392.358359:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434392.358364:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434392.358374:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913822592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434392.358477:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434392.358514:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434392.358551:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434393.255226:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434394.278038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434395.302003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434395.430907:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434395.430961:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434395.430976:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434395.431015:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434395.431024:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434395.431031:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789434395.431187:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434392/real 1789434395] req@ffff9811d0bdd880 x1876350913822464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434395.431211:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913822464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434395.431222:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434395.431226:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434395.431230:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434395.431237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434392/real 1789434395] req@ffff9811d0bdc000 x1876350913822592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434395.431247:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913822592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434395.431254:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434395.431257:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434395.431260:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822592:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789434395.431266:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434392/real 1789434395] req@ffff9811d0bdc700 x1876350913822208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434395.431277:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913822208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434395.431284:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434395.431287:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434395.431290:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913822208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434395.431295:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434392/real 1789434395] req@ffff9811d0bdea00 x1876350913822336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434395.431305:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913822336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434395.431311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434395.431314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434395.431317:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822336:192.168.203.128@tcp:38:kworker.0 00010000:00010000:0.0:1789434396.326011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434397.350284:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434397.481091:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434397.481101:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434397.481105:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434397.481115:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6767 00000100:00080000:3.0:1789434397.481124:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434397.481136:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434397.481178:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913822720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434397.481210:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434397.481215:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434397.481217:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434397.481221:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6767 00000100:00080000:3.0:1789434397.481225:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434397.481229:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434397.481239:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434397.481252:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913822848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434397.481263:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789434397.481268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913822720:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434397.481268:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434397.481272:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434397.481279:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6767 00000100:00080000:3.0:1789434397.481283:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434397.481288:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434397.481293:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434397.481303:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913822976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434397.481313:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434397.481316:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434397.481317:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434397.481321:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6767 00000100:00080000:3.0:1789434397.481324:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434397.481328:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434397.481332:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434397.481341:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913823104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434397.481445:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434397.481570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434397/real 1789434397] req@ffff9811d1bc7100 x1876350913822720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434397.481588:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913822720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434397.481597:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434397.481600:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434397.481604:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913822720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434397.481624:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434397.481648:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434397.481660:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434397/real 1789434397] req@ffff9811d1bc4700 x1876350913822848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434397.481671:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913822848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434397.481678:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434397.481681:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434397.481684:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434397.481692:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434397.481709:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434397.481828:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434397/real 1789434397] req@ffff9811d1bc6300 x1876350913822976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434397.481944:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913822976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434397.481951:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434397.481955:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434397.481958:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913822976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434397.481965:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434397.481986:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434397.481997:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434397/real 1789434397] req@ffff9811d1bc5500 x1876350913823104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434397.482009:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913823104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434397.482018:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434397.482021:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434397.482024:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434398.374043:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434399.398288:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434400.422237:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434401.446690:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434402.470034:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434402.597936:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434402.597943:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434402.597947:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434402.597955:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6772 00000100:00080000:3.0:1789434402.597961:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434402.597972:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434402.598006:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913823232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434402.598036:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434402.598039:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434402.598041:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434402.598045:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6772 00000100:00080000:3.0:1789434402.598048:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434402.598053:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434402.598062:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434402.598075:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913823232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434402.598076:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913823360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434402.598089:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434402.598092:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434402.598094:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434402.598098:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6772 00000100:00080000:3.0:1789434402.598102:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434402.598107:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434402.598112:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434402.598123:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913823488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434402.598132:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434402.598140:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434402.598144:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434402.598146:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434402.598151:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6772 00000100:00080000:3.0:1789434402.598164:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434402.598170:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434402.598175:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434402.598177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434402/real 1789434402] req@ffff9811d1bc5f80 x1876350913823232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434402.598191:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913823616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434402.598198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913823232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434402.598216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434402.598220:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434402.598225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913823232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434402.598247:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434402.598267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434402.598280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434402/real 1789434402] req@ffff9811d1bc4000 x1876350913823360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434402.598291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913823360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434402.598298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434402.598301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434402.598304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434402.598309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434402.598329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434402.598340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434402/real 1789434402] req@ffff9811d1bc5c00 x1876350913823488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434402.598357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913823488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434402.598364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434402.598367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434402.598369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434402.598375:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434402.598403:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434402.598418:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434402/real 1789434402] req@ffff9811d1bc6680 x1876350913823616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434402.598430:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913823616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434402.598438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434402.598441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434402.598444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434403.495118:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434404.518102:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434405.542295:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434406.566740:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434407.591346:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434407.717957:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434407.717966:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434407.717970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434407.717978:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6777 00000100:00080000:0.0:1789434407.717987:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434407.717998:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434407.718031:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913823744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434407.718058:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434407.718061:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434407.718063:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434407.718066:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6777 00000100:00080000:0.0:1789434407.718069:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434407.718073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434407.718082:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434407.718092:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913823872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434407.718101:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434407.718104:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434407.718106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434407.718110:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6777 00000100:00080000:0.0:1789434407.718114:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434407.718119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434407.718123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434407.718132:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913823744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434407.718135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913824000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434407.718153:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434407.718157:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434407.718159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434407.718163:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6777 00000100:00080000:0.0:1789434407.718166:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434407.718171:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434407.718177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434407.718186:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913824128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434407.718204:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434407.718296:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434407/real 1789434407] req@ffff9811c8cd7480 x1876350913823744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434407.718311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913823744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434407.718322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434407.718326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434407.718329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913823744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434407.718337:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434407.718361:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434407.718374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434407/real 1789434407] req@ffff9811c8cd7800 x1876350913823872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434407.718386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913823872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434407.718394:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434407.718397:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434407.718401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913823872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434407.718435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434407.718456:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434407.718467:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434407/real 1789434407] req@ffff9811c8cd5180 x1876350913824000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434407.718479:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913824000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434407.718487:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434407.718489:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434407.718492:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434407.718498:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824128:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434407.718518:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434407.718529:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434407/real 1789434407] req@ffff9811c8cd6a00 x1876350913824128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434407.718541:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913824128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434407.718549:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434407.718552:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434407.718554:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434408.614097:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434409.638013:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434410.662520:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434411.686006:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434412.710211:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434412.837985:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434412.837995:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434412.837999:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434412.838008:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6782 00000100:00080000:0.0:1789434412.838018:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434412.838031:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434412.838065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913824256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434412.838091:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434412.838094:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434412.838096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434412.838100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6782 00000100:00080000:0.0:1789434412.838103:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434412.838107:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434412.838117:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434412.838144:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913824256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434412.838145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913824384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434412.838162:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434412.838165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434412.838168:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434412.838171:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6782 00000100:00080000:0.0:1789434412.838175:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434412.838180:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434412.838185:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434412.838197:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913824512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434412.838207:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789434412.838219:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434412.838225:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434412.838228:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434412.838232:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6782 00000100:00080000:0.0:1789434412.838237:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434412.838242:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434412.838248:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434412.838262:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434412/real 1789434412] req@ffff9811c8cd4000 x1876350913824256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434412.838264:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913824640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434412.838295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913824256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434412.838306:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434412.838312:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434412.838316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913824256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434412.838336:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434412.838356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434412.838369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434412/real 1789434412] req@ffff9811c8cd6300 x1876350913824384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434412.838397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913824384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434412.838406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434412.838409:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434412.838412:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434412.838417:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434412.838435:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434412.838447:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434412/real 1789434412] req@ffff9811c8cd7b80 x1876350913824512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434412.838458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913824512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434412.838470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434412.838473:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434412.838476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434412.838481:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824640:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434412.838499:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434412.838508:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434412/real 1789434412] req@ffff9811c8cd4a80 x1876350913824640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434412.838517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913824640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434412.838537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434412.838548:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434412.838551:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434413.734285:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434414.758011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434415.782430:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434416.805996:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434417.830449:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434417.957958:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434417.957969:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434417.957973:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434417.957984:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6787 00000100:00080000:1.0:1789434417.957993:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434417.958004:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434417.958039:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913824768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434417.958070:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434417.958073:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434417.958075:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434417.958079:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6787 00000100:00080000:1.0:1789434417.958082:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434417.958087:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434417.958097:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434417.958114:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913824896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434417.958125:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789434417.958130:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913824768:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434417.958131:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434417.958134:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434417.958142:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6787 00000100:00080000:1.0:1789434417.958145:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434417.958149:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434417.958155:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434417.958172:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913825024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434417.958182:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434417.958186:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434417.958187:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434417.958192:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6787 00000800:00000100:3.0:1789434417.958196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434417.958198:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434417.958205:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434417.958249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434417/real 1789434417] req@ffff9811d0bdf800 x1876350913824768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434417.958268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913824768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434417.958278:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434417.958282:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434417.958285:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913824768:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434417.958306:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434417.958330:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434417.958361:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434417/real 1789434417] req@ffff9811d0bde300 x1876350913824896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434417.958375:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913824896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434417.958382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434417.958386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434417.958389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913824896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434417.958395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434417.958416:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434417.958426:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434417/real 1789434417] req@ffff9811d0bdf100 x1876350913825024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434417.958437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913825024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434417.958445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434417.958448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434417.958451:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825024:192.168.203.128@tcp:38:kworker.0 00000008:00100000:1.0:1789434417.959260:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434417.959276:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913825152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434417.959328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825152:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434417.959364:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434417.959377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434417/real 1789434417] req@ffff9811d0bdf100 x1876350913825152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434417.959391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913825152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434417.959400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434417.959403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434417.959407:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434418.853977:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434419.878000:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434420.902158:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434421.926045:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434422.950449:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434423.078018:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434423.078027:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434423.078031:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434423.078039:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6792 00000100:00080000:2.0:1789434423.078047:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434423.078055:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434423.078083:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913825280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434423.078104:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434423.078107:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434423.078108:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434423.078110:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6792 00000100:00080000:2.0:1789434423.078112:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434423.078114:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434423.078121:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434423.078129:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913825408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434423.078143:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434423.078146:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434423.078148:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434423.078151:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6792 00000100:00080000:2.0:1789434423.078154:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434423.078159:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434423.078163:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434423.078173:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913825536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434423.078183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434423.078190:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434423.078192:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434423.078195:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6792 00000100:00080000:2.0:1789434423.078198:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434423.078203:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434423.078207:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434423.078216:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913825664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434423.078797:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913825280:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789434423.078906:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434423.078955:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434423/real 1789434423] req@ffff9811d2101c00 x1876350913825280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434423.078976:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913825280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434423.078988:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434423.078993:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434423.078997:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913825280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434423.079004:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434423.079028:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434423.079041:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434423/real 1789434423] req@ffff9811d2103b80 x1876350913825408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434423.079053:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913825408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434423.079066:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434423.079070:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434423.079074:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434423.079080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434423.079100:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434423.079111:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434423/real 1789434423] req@ffff9811d2102a00 x1876350913825536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434423.079123:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913825536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434423.079131:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434423.079134:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434423.079137:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434423.079144:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434423.079165:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434423.079177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434423/real 1789434423] req@ffff9811d2101500 x1876350913825664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434423.079189:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913825664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434423.079199:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434423.079202:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434423.079207:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434423.974306:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434424.998246:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434426.022044:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434427.046082:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434428.069997:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434428.197973:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434428.197984:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434428.197998:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434428.198007:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6797 00000100:00080000:1.0:1789434428.198015:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434428.198034:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434428.198086:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913825792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434428.198107:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434428.198109:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434428.198110:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434428.198113:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6797 00000100:00080000:1.0:1789434428.198116:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434428.198120:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434428.198148:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434428.198158:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913825792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434428.198192:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913825920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789434428.198220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434428.198249:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434428/real 1789434428] req@ffff9811d0bde300 x1876350913825792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434428.198254:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434428.198259:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434428.198261:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789434428.198289:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913825792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434428.198299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434428.198304:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434428.198307:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913825792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434428.198329:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789434428.198349:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434428.198359:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434428/real 1789434428] req@ffff9811d0bdf800 x1876350913825920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434428.198371:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913825920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434428.198379:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434428.198382:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434428.198393:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913825920:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789434428.198423:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6797 00000100:00080000:1.0:1789434428.198429:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434428.198435:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434428.198450:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434428.198460:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913826048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434428.198485:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434428.198488:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434428.198490:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434428.198494:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6797 00000100:00080000:1.0:1789434428.198498:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789434428.198506:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826048:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789434428.198517:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789434428.198530:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434428.198540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434428/real 1789434428] req@ffff9811d0bdea00 x1876350913826048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789434428.198545:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434428.198551:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913826048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434428.198558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434428.198560:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434428.198562:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434428.198574:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913826176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789434428.198635:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789434428.198655:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434428.198666:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434428/real 1789434428] req@ffff9811d0bdc700 x1876350913826176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434428.198678:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913826176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434428.198687:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789434428.198690:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789434428.198693:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434429.094386:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434430.118779:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434431.142648:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434432.166689:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434433.190001:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434433.318292:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434433.318301:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434433.318305:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434433.318315:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6802 00000100:00080000:3.0:1789434433.318323:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434433.318335:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434433.318374:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913826304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434433.318457:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913826304:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434433.318521:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434433.318558:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434433.318562:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434433.318564:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434433.318569:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6802 00000100:00080000:3.0:1789434433.318573:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434433.318579:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434433.318589:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434433.318604:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913826432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434433.320272:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434433/real 1789434433] req@ffff9811d1bc5880 x1876350913826304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434433.320300:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913826304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434433.320313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434433.320318:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434433.320322:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913826304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434433.320361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434433.320417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434433.320434:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434433/real 1789434433] req@ffff9811d1bc7800 x1876350913826432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434433.320446:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913826432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434433.320455:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434433.320459:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434433.320462:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826432:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434433.320566:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434433.320572:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434433.320574:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434433.320579:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6802 00000100:00080000:3.0:1789434433.320585:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434433.320594:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434433.320606:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434433.320628:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913826560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434433.320642:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434433.320645:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434433.320646:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434433.320651:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6802 00000100:00080000:3.0:1789434433.320654:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434433.320659:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434433.320663:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434433.320673:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913826688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434433.320744:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434433.320768:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434433.320867:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434433/real 1789434433] req@ffff9811d1bc5180 x1876350913826560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434433.320880:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913826560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434433.320887:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434433.320891:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434433.320893:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434433.320899:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826688:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434433.320921:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434433.320931:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434433/real 1789434433] req@ffff9811d1bc6a00 x1876350913826688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434433.320942:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913826688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434433.320950:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434433.320952:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434433.320956:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434434.214601:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434435.239199:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434436.262418:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434437.287250:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434438.311018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434438.438674:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434438.438683:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434438.438687:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434438.438698:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6808 00000100:00080000:3.0:1789434438.438708:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434438.438719:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434438.438756:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913826816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434438.439642:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913826816:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434438.439705:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434438.439758:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434438.439763:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434438.439765:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434438.439770:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6808 00000100:00080000:3.0:1789434438.439775:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434438.439781:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434438.439791:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434438.439809:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913826944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434438.439954:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434438/real 1789434438] req@ffff9811d1bc7480 x1876350913826816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434438.439971:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913826816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434438.439981:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434438.439985:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434438.439988:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913826816:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434438.440110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434438.440114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434438.440116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434438.440121:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6808 00000100:00080000:3.0:1789434438.440126:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434438.440132:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434438.440138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434438.440148:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434438.440154:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913827072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434438.440168:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434438.440171:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434438.440173:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434438.440177:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6808 00000800:00000100:1.0:1789434438.440182:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434438.440195:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434438.440204:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434438.440210:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434438.440214:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434438/real 1789434438] req@ffff9811d1bc6a00 x1876350913826944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434438.440226:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913827200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434438.440230:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913826944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434438.440251:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434438.440254:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434438.440258:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913826944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434438.440271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434438.440287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434438.440298:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434438/real 1789434438] req@ffff9811d1bc5180 x1876350913827072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434438.440310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913827072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434438.440317:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434438.440321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434438.440324:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434438.440330:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434438.440346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434438.440354:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434438/real 1789434438] req@ffff9811d1bc7800 x1876350913827200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434438.440364:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913827200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434438.440371:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434438.440373:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434438.440376:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434439.334161:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434440.358263:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434441.382032:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434442.405992:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434443.430588:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434443.557989:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434443.558000:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434443.558005:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434443.558017:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6813 00000100:00080000:3.0:1789434443.558029:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434443.558043:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434443.558088:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913827328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434443.558125:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434443.558128:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434443.558130:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434443.558135:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6813 00000100:00080000:3.0:1789434443.558138:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434443.558143:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434443.558161:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434443.558173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913827456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434443.558186:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434443.558190:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434443.558192:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434443.558195:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6813 00000100:00080000:3.0:1789434443.558198:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434443.558203:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434443.558208:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434443.558217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913827584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434443.558227:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434443.558231:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434443.558233:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434443.558237:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6813 00000100:00080000:3.0:1789434443.558241:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434443.558245:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434443.558250:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434443.558259:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913827712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434443.558366:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913827328:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434443.558441:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434443.558505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434443/real 1789434443] req@ffff9811d1bc5880 x1876350913827328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434443.558527:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913827328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434443.558539:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434443.558544:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434443.558548:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913827328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434443.558555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434443.558577:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434443.558589:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434443/real 1789434443] req@ffff9811d1bc6680 x1876350913827456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434443.558600:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913827456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434443.558606:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434443.558610:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434443.558612:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434443.558618:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434443.558635:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434443.558645:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434443/real 1789434443] req@ffff9811d1bc5c00 x1876350913827584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434443.559485:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913827584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434443.559494:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434443.559497:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434443.559501:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434443.559508:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434443.559538:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434443.559550:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434443/real 1789434443] req@ffff9811d1bc4000 x1876350913827712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434443.559563:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913827712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434443.559571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434443.559575:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434443.559578:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434444.454108:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434445.478260:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434446.502363:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434447.526276:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434448.550390:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434448.677912:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434448.677922:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434448.677925:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434448.677934:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6818 00000100:00080000:3.0:1789434448.677942:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434448.677953:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434448.677987:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913827840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434448.678019:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434448.678023:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434448.678024:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434448.678028:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6818 00000100:00080000:3.0:1789434448.678031:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434448.678035:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434448.678045:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434448.678056:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913827968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434448.678064:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434448.678069:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434448.678071:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434448.678074:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6818 00000100:00080000:3.0:1789434448.678077:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434448.678081:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434448.678085:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434448.678094:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913828096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434448.678102:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434448.678105:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434448.678106:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434448.678109:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6818 00000100:00080000:3.0:1789434448.678126:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434448.678129:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434448.678134:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434448.678144:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913828224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434448.678221:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913827840:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434448.678278:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434448.678311:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434448/real 1789434448] req@ffff9811d1bc5f80 x1876350913827840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434448.678326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913827840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434448.678335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434448.678338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434448.678341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913827840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434448.678347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434448.678367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434448.678378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434448/real 1789434448] req@ffff9811d1bc5500 x1876350913827968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434448.678388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913827968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434448.678395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434448.678398:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434448.678400:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913827968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434448.678404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434448.678419:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434448.678429:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434448/real 1789434448] req@ffff9811d1bc6300 x1876350913828096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434448.678438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913828096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434448.678444:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434448.678446:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434448.678449:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434448.678454:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828224:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434448.678471:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434448.678481:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434448/real 1789434448] req@ffff9811d1bc4700 x1876350913828224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434448.678491:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913828224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434448.678499:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434448.678502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434448.678504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434449.574089:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434450.598218:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434451.622137:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434452.646021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434453.670278:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434453.798012:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434453.798022:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434453.798026:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434453.798035:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6823 00000100:00080000:3.0:1789434453.798045:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434453.798056:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434453.798110:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913828352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434453.798136:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434453.798141:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434453.798143:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434453.798147:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6823 00000100:00080000:3.0:1789434453.798151:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434453.798156:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434453.798165:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434453.798179:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913828480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434453.798188:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434453.798192:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434453.798194:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434453.798197:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6823 00000100:00080000:3.0:1789434453.798200:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434453.798205:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434453.798209:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434453.798221:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913828608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434453.798227:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913828352:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434453.798234:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434453.798239:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434453.798240:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434453.798244:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6823 00000100:00080000:3.0:1789434453.798249:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434453.798254:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434453.798259:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434453.798273:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913828736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434453.798287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434453.798323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434453/real 1789434453] req@ffff9811d1bc7100 x1876350913828352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434453.798341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913828352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434453.798351:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434453.798355:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434453.798369:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913828352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434453.798377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434453.798399:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434453.798410:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434453/real 1789434453] req@ffff9811d1bc4700 x1876350913828480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434453.798432:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913828480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434453.798446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434453.798457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434453.798460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434453.798481:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434453.798500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434453.798511:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434453/real 1789434453] req@ffff9811d1bc6300 x1876350913828608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434453.798521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913828608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434453.798528:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434453.798532:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434453.798534:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434453.798541:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434453.798557:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434453.798566:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434453/real 1789434453] req@ffff9811d1bc5500 x1876350913828736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434453.798577:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913828736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434453.798584:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434453.798587:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434453.798590:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434454.694116:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434455.718103:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434456.742087:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434457.766419:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434458.791164:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434458.918518:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434458.918525:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434458.918529:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434458.918537:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6828 00000100:00080000:2.0:1789434458.918543:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434458.918552:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434458.918618:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913828864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434458.918639:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434458.918641:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434458.918643:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434458.918645:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6828 00000100:00080000:2.0:1789434458.918648:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434458.918651:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434458.918659:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434458.918666:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913828992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434458.918672:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434458.918675:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434458.918676:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434458.918678:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6828 00000100:00080000:2.0:1789434458.918680:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434458.918682:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434458.918684:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434458.918691:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913829120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434458.918700:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434458.918703:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434458.918705:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434458.918708:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6828 00000100:00080000:2.0:1789434458.918711:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434458.918715:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434458.918720:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434458.918732:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913829248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434458.918816:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913828864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434458.918973:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434458.918997:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434458.919011:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434459.814336:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434460.838140:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434461.862100:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789434461.990273:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789434461.990336:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789434461.990355:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434461.990395:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434461.990406:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434461.990418:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434461.990420:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434458/real 1789434461] req@ffff9811d2102d80 x1876350913828864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434461.990450:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913828864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434461.990464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434461.990470:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434461.990475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913828864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434461.990484:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434458/real 1789434461] req@ffff9811d2103800 x1876350913828992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434461.990496:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913828992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434461.990503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434461.990507:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434461.990511:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913828992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434461.990518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434458/real 1789434461] req@ffff9811d2100000 x1876350913829120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434461.990529:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913829120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434461.990536:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434461.990539:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434461.990542:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434461.990548:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434458/real 1789434461] req@ffff9811d2101500 x1876350913829248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434461.990558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913829248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434461.990566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434461.990569:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434461.990573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434462.886095:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434463.912322:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434464.038139:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434464.038149:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434464.038152:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434464.038161:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6833 00000100:00080000:2.0:1789434464.038170:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434464.038180:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434464.038210:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913829376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434464.038417:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913829376:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434464.038473:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434464.038504:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434464.038509:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434464.038511:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434464.038515:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6833 00000100:00080000:2.0:1789434464.038520:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434464.038525:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434464.038535:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434464.038548:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913829504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434464.038733:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434464/real 1789434464] req@ffff9811d2102a00 x1876350913829376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434464.038750:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913829376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434464.038761:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434464.038764:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434464.038767:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913829376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434464.038789:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434464.038812:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434464.038825:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434464/real 1789434464] req@ffff9811d2103b80 x1876350913829504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434464.038867:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913829504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434464.038876:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434464.038879:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434464.038882:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829504:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434464.040144:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434464.040151:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434464.040154:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434464.040162:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6833 00000100:00080000:2.0:1789434464.040170:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434464.040180:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434464.040191:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434464.040216:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913829632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434464.040231:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434464.040234:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434464.040236:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434464.040240:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6833 00000100:00080000:2.0:1789434464.040904:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434464.040914:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434464.040922:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434464.040939:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913829760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434464.041123:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434464.041172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434464.041197:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434464/real 1789434464] req@ffff9811d2101c00 x1876350913829632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434464.041211:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913829632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434464.041219:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434464.041222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434464.041225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434464.041231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434464.041589:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434464.041604:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434464/real 1789434464] req@ffff9811d2101880 x1876350913829760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434464.041616:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913829760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434464.041624:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434464.041627:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434464.041631:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913829760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434464.934195:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434465.958132:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434466.982403:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434468.006871:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434469.030481:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434469.158078:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434469.158088:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434469.158093:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434469.158103:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6838 00000100:00080000:0.0:1789434469.158112:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434469.158123:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434469.158164:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913829888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434469.158196:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434469.158200:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434469.158202:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434469.158206:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6838 00000100:00080000:0.0:1789434469.158209:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434469.158214:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434469.158223:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434469.158236:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913830016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434469.158245:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434469.158248:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434469.158250:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434469.158254:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6838 00000100:00080000:0.0:1789434469.158257:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434469.158260:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434469.158265:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434469.158287:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913830144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434469.158297:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434469.158300:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434469.158302:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434469.158306:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6838 00000100:00080000:0.0:1789434469.158309:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434469.158313:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434469.158318:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434469.158330:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913830272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434469.158417:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913829888:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434469.158476:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434469.158513:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434469/real 1789434469] req@ffff9811c8cd5500 x1876350913829888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434469.158528:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913829888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434469.158538:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434469.158542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434469.158546:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913829888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434469.158553:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434469.158573:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434469.158584:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434469/real 1789434469] req@ffff9811c8cd6d80 x1876350913830016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434469.158595:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913830016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434469.158603:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434469.158606:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434469.158609:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434469.158614:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434469.158631:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434469.158642:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434469/real 1789434469] req@ffff9811c8cd5f80 x1876350913830144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434469.158652:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913830144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434469.158659:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434469.158662:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434469.158665:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434469.158670:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434469.158688:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434469.158696:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434469/real 1789434469] req@ffff9811c8cd4a80 x1876350913830272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434469.158705:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913830272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434469.158712:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434469.158715:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434469.158717:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434470.054140:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434471.078099:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434472.102267:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434473.126210:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434474.150146:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434474.277978:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434474.277988:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434474.277992:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434474.278003:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6843 00000100:00080000:0.0:1789434474.278014:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434474.278024:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434474.278066:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913830400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434474.278103:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434474.278107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434474.278109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434474.278113:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6843 00000100:00080000:0.0:1789434474.278116:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434474.278121:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434474.278130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434474.278145:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913830528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434474.278157:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434474.278168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434474.278171:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434474.278177:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6843 00000100:00080000:0.0:1789434474.278182:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434474.278192:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434474.278200:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913830400:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789434474.278202:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434474.278225:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913830656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434474.278238:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434474.278246:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434474.278248:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434474.278252:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6843 00000100:00080000:0.0:1789434474.278255:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434474.278275:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434474.278280:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789434474.278292:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434474.278294:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913830784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434474.278348:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434474/real 1789434474] req@ffff9811c8cd7b80 x1876350913830400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434474.278368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913830400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434474.278380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434474.278384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434474.278389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913830400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434474.278397:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434474.278420:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434474.278432:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434474/real 1789434474] req@ffff9811c8cd6300 x1876350913830528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434474.278444:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913830528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434474.278453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434474.278456:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434474.278459:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434474.278484:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434474.278501:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434474.278511:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434474/real 1789434474] req@ffff9811c8cd4000 x1876350913830656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434474.278522:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913830656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434474.278529:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434474.278533:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434474.278535:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434474.278541:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830784:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434474.278559:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434474.278568:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434474/real 1789434474] req@ffff9811c8cd6a00 x1876350913830784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434474.278579:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913830784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434474.278592:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434474.278595:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434474.278597:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913830784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434475.175531:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434476.198051:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434477.222006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434478.246035:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434479.270061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434479.397972:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434479.397981:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434479.397986:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434479.397997:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6848 00000100:00080000:3.0:1789434479.398006:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434479.398018:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434479.398065:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913830912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434479.398095:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434479.398099:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434479.398101:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434479.398105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6848 00000100:00080000:3.0:1789434479.398109:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434479.398114:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434479.398123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434479.398135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913831040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434479.398146:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434479.398152:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434479.398153:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434479.398156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6848 00000100:00080000:3.0:1789434479.398159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434479.398164:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434479.398168:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434479.398177:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913831168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434479.398187:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434479.398190:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434479.398191:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434479.398194:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6848 00000100:00080000:3.0:1789434479.398198:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434479.398201:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434479.398206:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434479.398217:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913831296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434479.398269:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913830912:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434479.398335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434479.398389:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434479/real 1789434479] req@ffff9811d1bc5f80 x1876350913830912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434479.398404:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913830912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434479.398414:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434479.398416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434479.398420:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913830912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434479.398426:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434479.398458:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434479.398471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434479/real 1789434479] req@ffff9811d1bc4000 x1876350913831040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434479.398489:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913831040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434479.398497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434479.398502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434479.398504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434479.398510:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434479.398529:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434479.398540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434479/real 1789434479] req@ffff9811d1bc5c00 x1876350913831168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434479.398550:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913831168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434479.398556:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434479.398559:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434479.398562:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434479.398568:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434479.398585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434479.398594:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434479/real 1789434479] req@ffff9811d1bc6680 x1876350913831296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434479.398604:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913831296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434479.398611:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434479.398614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434479.398618:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434480.294075:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434481.318177:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434482.342048:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434483.367167:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434484.390023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434484.517975:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434484.517988:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434484.517992:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434484.518005:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6854 00000100:00080000:0.0:1789434484.518014:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434484.518025:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434484.518067:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913831424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434484.518096:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434484.518100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434484.518102:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434484.518107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6854 00000100:00080000:0.0:1789434484.518110:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434484.518115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434484.518125:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434484.518139:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913831552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434484.518152:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434484.518155:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434484.518157:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789434484.518161:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913831424:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434484.518163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6854 00000100:00080000:0.0:1789434484.518169:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434484.518175:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434484.518180:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434484.518192:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913831680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434484.518203:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434484.518206:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434484.518208:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434484.518211:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6854 00000100:00080000:0.0:1789434484.518215:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434484.518220:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789434484.518230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789434484.518235:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434484.518250:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913831808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434484.518275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434484/real 1789434484] req@ffff9811c8cd5180 x1876350913831424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434484.518307:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913831424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434484.518328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434484.518331:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434484.518334:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913831424:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434484.518352:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434484.518368:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434484.518377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434484/real 1789434484] req@ffff9811c8cd7800 x1876350913831552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434484.518386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913831552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434484.518391:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434484.518393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434484.518395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434484.518398:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434484.518409:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434484.518419:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434484/real 1789434484] req@ffff9811c8cd7480 x1876350913831680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434484.518426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913831680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434484.518435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434484.518437:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434484.518439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434484.518442:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831808:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434484.518455:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434484.518464:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434484/real 1789434484] req@ffff9811c8cd7100 x1876350913831808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434484.518474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913831808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434484.518482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434484.518485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434484.518488:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913831808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434485.414122:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434486.438063:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434487.462161:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434488.486387:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434489.510023:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434489.638454:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434489.638464:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434489.638468:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434489.638478:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6859 00000100:00080000:2.0:1789434489.638485:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434489.638497:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434489.638533:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913831936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434489.638565:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434489.638568:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434489.638569:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434489.638573:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6859 00000100:00080000:2.0:1789434489.638577:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434489.638582:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434489.638590:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434489.638601:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913832064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434489.638610:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434489.638613:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434489.638615:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434489.638619:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6859 00000100:00080000:2.0:1789434489.638622:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434489.638626:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434489.638631:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434489.638640:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913832192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434489.638650:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434489.638653:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434489.638654:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434489.638658:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6859 00000100:00080000:2.0:1789434489.638661:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434489.638665:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434489.638669:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434489.638680:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913832320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434489.638764:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913831936:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434489.638854:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434489.638904:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434489/real 1789434489] req@ffff9811d2101180 x1876350913831936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434489.638922:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913831936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434489.638931:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434489.638934:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434489.638938:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913831936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434489.638945:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434489.638968:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434489.638980:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434489/real 1789434489] req@ffff9811d2101f80 x1876350913832064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434489.638992:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913832064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434489.639000:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434489.639003:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434489.639006:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434489.639012:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434489.639030:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434489.639041:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434489/real 1789434489] req@ffff9811d2100e00 x1876350913832192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434489.639053:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913832192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434489.639061:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434489.639064:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434489.639067:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434489.639073:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434489.639092:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434489.639104:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434489/real 1789434489] req@ffff9811d2101880 x1876350913832320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434489.639115:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913832320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434489.639122:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434489.639125:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434489.639128:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434490.534193:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434491.558194:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434492.582150:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434493.606006:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434494.630084:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434494.758174:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434494.758184:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434494.758188:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434494.758197:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6864 00000100:00080000:3.0:1789434494.758205:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434494.758216:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434494.758250:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913832448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434494.758278:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434494.758281:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434494.758283:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434494.758286:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6864 00000100:00080000:3.0:1789434494.758290:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434494.758294:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434494.758304:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434494.758319:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913832576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434494.758328:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434494.758330:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434494.758331:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434494.758333:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6864 00000100:00080000:3.0:1789434494.758336:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434494.758339:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434494.758342:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434494.758348:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913832704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434494.758354:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434494.758356:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434494.758357:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434494.758359:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6864 00000100:00080000:3.0:1789434494.758361:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434494.758372:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434494.758379:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434494.758391:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913832832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434494.758436:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913832448:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434494.758494:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434494.758540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434494/real 1789434494] req@ffff9811d1bc5880 x1876350913832448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434494.758557:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913832448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434494.758566:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434494.758570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434494.758574:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913832448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434494.758580:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434494.758601:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434494.758613:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434494/real 1789434494] req@ffff9811d1bc7800 x1876350913832576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434494.758624:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913832576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434494.758647:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434494.758651:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434494.758655:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434494.758662:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434494.758681:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434494.758692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434494/real 1789434494] req@ffff9811d1bc5180 x1876350913832704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434494.758703:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913832704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434494.758710:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434494.758713:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434494.758717:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434494.758723:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434494.758741:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434494.758751:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434494/real 1789434494] req@ffff9811d1bc6a00 x1876350913832832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434494.758763:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913832832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434494.758783:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434494.758786:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434494.758790:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913832832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434495.654923:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434496.678077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434497.702035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434498.726066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434499.750002:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434499.877943:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434499.877952:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434499.877957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434499.877967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6869 00000100:00080000:3.0:1789434499.877975:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434499.877986:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434499.878020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913832960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434499.878045:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434499.878047:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434499.878048:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434499.878052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6869 00000100:00080000:3.0:1789434499.878055:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434499.878060:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434499.878071:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434499.878080:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913832960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434499.878087:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913833088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434499.878099:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434499.878102:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434499.878104:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434499.878107:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6869 00000100:00080000:3.0:1789434499.878111:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434499.878115:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434499.878120:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434499.878131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913833216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434499.878135:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434499.878145:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434499.878149:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434499.878151:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434499.878155:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6869 00000100:00080000:3.0:1789434499.878158:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434499.878162:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434499.878167:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434499.878177:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434499/real 1789434499] req@ffff9811d1bc7480 x1876350913832960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434499.878180:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913833344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434499.878213:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913832960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434499.878224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434499.878228:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434499.878231:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913832960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434499.878249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434499.878268:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434499.878283:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434499/real 1789434499] req@ffff9811d1bc6a00 x1876350913833088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434499.878295:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913833088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434499.878300:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434499.878302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434499.878304:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434499.878309:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434499.878326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434499.878338:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434499/real 1789434499] req@ffff9811d1bc5180 x1876350913833216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434499.878349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913833216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434499.878357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434499.878360:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434499.878363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434499.878368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833344:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434499.878387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434499.878398:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434499/real 1789434499] req@ffff9811d1bc7800 x1876350913833344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434499.878409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913833344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434499.878417:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434499.878420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434499.878423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434500.774011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434501.797995:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434502.822124:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434503.846077:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434504.870037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434504.997909:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434504.997919:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434504.997923:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434504.997934:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6874 00000100:00080000:3.0:1789434504.997954:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434504.997967:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434504.998012:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913833472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434504.998042:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434504.998046:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434504.998048:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434504.998052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6874 00000100:00080000:3.0:1789434504.998056:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434504.998061:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434504.998080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434504.998091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913833472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434504.998097:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913833600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434504.998109:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434504.998113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434504.998116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434504.998119:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6874 00000100:00080000:3.0:1789434504.998124:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434504.998129:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434504.998134:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789434504.998143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434504.998147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913833728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434504.998156:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434504.998159:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434504.998161:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434504.998166:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6874 00000100:00080000:3.0:1789434504.998170:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434504.998175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434504.998180:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434504.998184:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434504/real 1789434504] req@ffff9811d1bc5880 x1876350913833472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434504.998196:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913833856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434504.998206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913833472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434504.998218:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434504.998222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434504.998227:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913833472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434504.998248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434504.998271:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434504.998284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434504/real 1789434504] req@ffff9811d1bc6680 x1876350913833600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434504.998297:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913833600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434504.998304:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434504.998308:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434504.998311:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434504.998317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434504.998335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434504.998345:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434504/real 1789434504] req@ffff9811d1bc5c00 x1876350913833728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434504.998357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913833728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434504.998364:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434504.998367:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434504.998370:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833728:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434504.998376:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434504.998395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434504.998406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434504/real 1789434504] req@ffff9811d1bc4000 x1876350913833856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434504.998416:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913833856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434504.998424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434504.998427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434504.998430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913833856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434505.894054:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434506.918002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434507.942452:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434508.966152:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434509.990009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434510.117950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434510.117964:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434510.117969:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434510.117980:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6879 00000100:00080000:3.0:1789434510.117989:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434510.118000:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434510.118033:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913833984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434510.118064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434510.118068:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434510.118069:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434510.118074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6879 00000100:00080000:3.0:1789434510.118078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434510.118083:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434510.118093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434510.118111:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913834112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434510.118120:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434510.118123:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434510.118125:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434510.118128:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6879 00000100:00080000:3.0:1789434510.118132:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434510.118137:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434510.118142:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434510.118155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913834240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434510.118165:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434510.118168:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434510.118170:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434510.118173:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6879 00000100:00080000:3.0:1789434510.118176:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434510.118181:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434510.118186:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434510.118199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913834368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434510.118205:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913833984:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434510.118276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434510.118313:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434510/real 1789434510] req@ffff9811d1bc5f80 x1876350913833984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434510.118330:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913833984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434510.118341:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434510.118345:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434510.118348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913833984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434510.118359:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434510.118381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434510.118392:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434510/real 1789434510] req@ffff9811d1bc5500 x1876350913834112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434510.118404:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913834112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434510.118412:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434510.118416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434510.118418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434510.118424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434510.118454:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434510.118464:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434510/real 1789434510] req@ffff9811d1bc6300 x1876350913834240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434510.118481:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913834240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434510.118488:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434510.118491:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434510.118494:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434510.118518:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434510.118538:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434510.118548:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434510/real 1789434510] req@ffff9811d1bc4700 x1876350913834368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434510.118560:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913834368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434510.118567:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434510.118570:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434510.118573:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434511.014025:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434512.038011:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434513.062011:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434514.087027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434515.110093:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434515.239763:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434515.239772:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434515.239776:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434515.239786:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6884 00000100:00080000:0.0:1789434515.239795:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434515.239805:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434515.239859:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913834496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434515.239889:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434515.239893:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434515.239895:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434515.239899:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6884 00000100:00080000:0.0:1789434515.239903:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434515.239907:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434515.239917:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434515.239935:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913834624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434515.239945:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434515.239948:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434515.239950:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434515.239953:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6884 00000100:00080000:0.0:1789434515.239956:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434515.239960:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434515.239964:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434515.239973:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913834752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434515.239983:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434515.239986:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434515.239987:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434515.239991:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6884 00000100:00080000:0.0:1789434515.239995:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434515.240000:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434515.240004:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434515.240014:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913834496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434515.240020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913834880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434515.240103:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434515.240153:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434515/real 1789434515] req@ffff9811c8cd5c00 x1876350913834496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434515.240171:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913834496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434515.240184:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434515.240188:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434515.240192:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913834496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434515.240200:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434515.240286:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434515.240301:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434515/real 1789434515] req@ffff9811c8cd4700 x1876350913834624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434515.240314:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913834624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434515.240322:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434515.240325:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434515.240329:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434515.240335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434515.240354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434515.240365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434515/real 1789434515] req@ffff9811c8cd6680 x1876350913834752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434515.240377:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913834752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434515.240384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434515.240387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434515.240389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434515.240418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834880:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434515.240439:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434515.240449:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434515/real 1789434515] req@ffff9811c8cd7100 x1876350913834880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434515.240460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913834880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434515.240473:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434515.240476:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434515.240479:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913834880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434516.135565:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434517.158173:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434518.182933:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434519.206233:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434520.230079:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434520.357959:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434520.357969:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434520.357973:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434520.357982:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6889 00000100:00080000:0.0:1789434520.357992:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434520.358002:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434520.358036:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913835008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434520.358068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434520.358072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434520.358074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434520.358079:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6889 00000100:00080000:0.0:1789434520.358083:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434520.358088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434520.358096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434520.358108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913835136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434520.358118:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434520.358121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434520.358123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434520.358126:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6889 00000100:00080000:0.0:1789434520.358129:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434520.358132:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434520.358135:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434520.358149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913835264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434520.358161:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434520.358164:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434520.358165:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434520.358168:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6889 00000100:00080000:0.0:1789434520.358171:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434520.358175:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434520.358178:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434520.358191:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913835392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434520.358251:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913835008:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434520.358359:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434520.358416:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434520/real 1789434520] req@ffff9811c8cd7480 x1876350913835008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434520.358448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913835008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434520.358460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434520.358465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434520.358470:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913835008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434520.358479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434520.358504:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434520.358518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434520/real 1789434520] req@ffff9811c8cd7800 x1876350913835136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434520.358530:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913835136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434520.358549:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434520.358553:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434520.358556:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434520.358563:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434520.358595:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434520.358608:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434520/real 1789434520] req@ffff9811c8cd5180 x1876350913835264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434520.358619:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913835264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434520.358626:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434520.358630:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434520.358634:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434520.358640:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434520.358663:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434520.358674:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434520/real 1789434520] req@ffff9811c8cd6a00 x1876350913835392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434520.358686:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913835392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434520.358694:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434520.358697:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434520.358700:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434521.254028:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434522.281052:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434523.302108:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434524.326169:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434525.351766:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434525.478003:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434525.478013:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434525.478046:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434525.478057:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6895 00000100:00080000:2.0:1789434525.478068:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434525.478079:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434525.478113:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913835520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434525.478200:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913835520:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434525.478298:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434525.478303:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434525.478304:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434525.478309:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6895 00000100:00080000:2.0:1789434525.478313:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434525.478317:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434525.478325:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434525.478335:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913835648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434525.478458:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835648:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434525.478574:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434525.478580:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434525.478582:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434525.478590:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6895 00000100:00080000:2.0:1789434525.478598:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434525.478608:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434525.478617:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434525.478635:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913835776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434525.478656:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434525.478660:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434525.478664:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434525.478673:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6895 00000100:00080000:2.0:1789434525.478678:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789434525.478691:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835776:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434525.478792:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434525.478803:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434525.478820:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913835904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434525.478969:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434526.375759:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434527.399294:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434528.422037:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789434528.550104:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:3.0:1789434528.550145:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789434528.550161:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789434528.550202:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789434528.550210:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:3.0:1789434528.550218:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434528.550412:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434525/real 1789434528] req@ffff9811d2101500 x1876350913835904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434528.550438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913835904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434528.550450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434528.550454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434528.550458:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835904:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789434528.550465:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434525/real 1789434528] req@ffff9811d2102a00 x1876350913835776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434528.550477:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913835776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434528.550485:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434528.550489:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434528.550493:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434528.550500:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434525/real 1789434528] req@ffff9811d2103b80 x1876350913835648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434528.550511:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913835648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434528.550535:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434528.550538:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434528.550544:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913835648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434528.550551:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434525/real 1789434528] req@ffff9811d2101c00 x1876350913835520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434528.550564:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913835520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434528.550571:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434528.550575:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434528.550577:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913835520:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789434529.446027:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434530.470557:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434530.597962:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434530.597973:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434530.597978:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434530.597988:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6900 00000100:00080000:0.0:1789434530.597995:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434530.598006:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434530.598037:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913836032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434530.598071:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434530.598075:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434530.598077:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434530.598080:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6900 00000100:00080000:0.0:1789434530.598084:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434530.598089:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434530.598099:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434530.598112:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913836160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434530.598116:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913836032:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434530.598129:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434530.598133:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434530.598134:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434530.598150:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6900 00000100:00080000:0.0:1789434530.598154:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434530.598168:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434530.598173:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789434530.598180:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434530.598187:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913836288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434530.598203:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434530.598207:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434530.598208:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434530.598212:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6900 00000100:00080000:0.0:1789434530.598216:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789434530.598218:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434530/real 1789434530] req@ffff9811c8cd4000 x1876350913836032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434530.598223:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434530.598231:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434530.598241:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913836032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434530.598246:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913836416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434530.598250:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434530.598258:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434530.598264:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913836032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434530.598288:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434530.598312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434530.598324:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434530/real 1789434530] req@ffff9811c8cd6300 x1876350913836160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434530.598333:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913836160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434530.598341:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434530.598344:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434530.598347:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434530.598353:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434530.598370:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434530.598381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434530/real 1789434530] req@ffff9811c8cd7b80 x1876350913836288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434530.598391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913836288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434530.598399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434530.598402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434530.598405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434530.598411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434530.598430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434530.598440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434530/real 1789434530] req@ffff9811c8cd4a80 x1876350913836416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434530.598458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913836416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434530.598466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434530.598468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434530.598471:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434531.494022:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434532.518002:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434533.542404:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434534.566009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434535.590009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434535.718899:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434535.718908:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434535.718912:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434535.718923:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6905 00000100:00080000:0.0:1789434535.718930:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434535.718941:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434535.718988:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913836544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434535.720566:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434535.720572:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434535.720574:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434535.720585:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6905 00000100:00100000:1.0:1789434535.720587:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913836544:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434535.720595:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434535.720605:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434535.720626:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434535.720646:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913836672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434535.720655:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434535.720666:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434535.720669:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434535.720670:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434535.720674:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6905 00000100:00080000:0.0:1789434535.720677:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434535.720681:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434535.720685:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434535.720694:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913836800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434535.720703:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434535/real 1789434535] req@ffff9811c8cd5f80 x1876350913836544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434535.720704:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434535.720711:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434535.720713:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434535.720721:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6905 00000100:00080000:0.0:1789434535.720724:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434535.720728:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434535.720732:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434535.720734:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913836544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434535.720743:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913836928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434535.720748:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434535.720764:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434535.720769:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913836544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434535.720793:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434535.720816:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434535.720829:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434535/real 1789434535] req@ffff9811c8cd6d80 x1876350913836672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434535.720883:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913836672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434535.720898:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434535.720901:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434535.720904:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434535.720910:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434535.720930:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434535.720942:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434535/real 1789434535] req@ffff9811c8cd5500 x1876350913836800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434535.720954:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913836800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434535.720961:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434535.720972:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434535.720978:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434535.720984:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434535.721005:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434535.721027:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434535/real 1789434535] req@ffff9811c8cd4a80 x1876350913836928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434535.721041:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913836928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434535.721048:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434535.721052:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434535.721068:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913836928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434536.614503:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434537.638963:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434538.662148:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434539.686053:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434540.710237:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434540.837995:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434540.838004:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434540.838007:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434540.838015:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6910 00000100:00080000:2.0:1789434540.838022:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434540.838031:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434540.838068:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913837056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434540.838096:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434540.838100:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434540.838102:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434540.838106:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6910 00000100:00080000:2.0:1789434540.838109:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434540.838114:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434540.838124:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434540.838136:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913837184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434540.838144:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913837056:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434540.838149:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434540.838153:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434540.838155:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434540.838158:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6910 00000100:00080000:2.0:1789434540.838162:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434540.838167:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434540.838171:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434540.838182:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913837312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434540.838191:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434540.838194:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434540.838196:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434540.838199:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6910 00000800:00000100:1.0:1789434540.838203:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434540.838205:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434540.838211:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434540.838216:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434540.838228:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913837440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434540.838237:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434540/real 1789434540] req@ffff9811d2100000 x1876350913837056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434540.838254:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913837056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434540.838264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434540.838269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434540.838273:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913837056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434540.838296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434540.838319:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434540.838332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434540/real 1789434540] req@ffff9811d2103800 x1876350913837184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434540.838346:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913837184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434540.838355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434540.838359:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434540.838363:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434540.838369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434540.838390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434540.838402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434540/real 1789434540] req@ffff9811d2102d80 x1876350913837312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434540.838413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913837312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434540.838419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434540.838422:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434540.838425:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434540.838449:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837440:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434540.838469:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434540.838479:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434540/real 1789434540] req@ffff9811d2101c00 x1876350913837440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434540.838490:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913837440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434540.838498:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434540.838502:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434540.838505:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434541.734181:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434542.758241:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434543.782042:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434544.806409:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434545.830037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434545.957904:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434545.957918:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434545.957922:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434545.957932:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6915 00000100:00080000:2.0:1789434545.957941:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434545.957952:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434545.957982:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913837568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434545.958013:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434545.958017:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434545.958019:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434545.958026:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6915 00000100:00100000:1.0:1789434545.958162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913837568:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434545.958221:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434545.958255:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434545/real 1789434545] req@ffff9811d2103b80 x1876350913837568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434545.958272:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913837568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434545.958283:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434545.958286:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434545.958290:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913837568:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434545.958374:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434545.958381:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434545.958405:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434545.958418:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913837696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434545.958438:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434545.958441:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434545.958443:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434545.958446:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6915 00000100:00080000:2.0:1789434545.958449:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434545.958453:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434545.958457:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434545.958465:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913837824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434545.958486:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434545.958488:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434545.958490:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434545.958492:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6915 00000100:00080000:2.0:1789434545.958495:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434545.958499:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434545.958504:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434545.958522:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913837952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434545.959363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434545.959822:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434545.959875:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434545/real 1789434545] req@ffff9811d2102a00 x1876350913837696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434545.959892:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913837696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434545.959902:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434545.959907:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434545.959910:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434545.959917:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434545.959940:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434545.959952:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434545/real 1789434545] req@ffff9811d2101500 x1876350913837824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434545.959962:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913837824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434545.959970:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434545.959974:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434545.959977:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434545.959983:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434545.960002:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434545.960013:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434545/real 1789434545] req@ffff9811d2101880 x1876350913837952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434545.960023:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913837952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434545.960031:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434545.960034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434545.960037:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913837952:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434546.854286:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434547.879437:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434548.903431:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434549.926407:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434550.950058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434551.078000:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434551.078008:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434551.078012:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434551.078022:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6920 00000100:00080000:0.0:1789434551.078030:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434551.078042:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434551.078074:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913838080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434551.078145:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913838080:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434551.078210:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434551.078298:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434551.078302:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434551.078304:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434551.078308:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6920 00000100:00080000:0.0:1789434551.078312:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434551.078317:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434551.078327:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434551.078340:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913838208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434551.078408:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434551/real 1789434551] req@ffff9811c8cd7b80 x1876350913838080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434551.078426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913838080/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434551.078435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434551.078439:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434551.078443:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913838080:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434551.078465:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434551.078487:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434551.078499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434551/real 1789434551] req@ffff9811c8cd6300 x1876350913838208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434551.078512:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913838208/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434551.078519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434551.078522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434551.078525:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838208:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434551.078651:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434551.078655:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434551.078658:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434551.078662:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6920 00000100:00080000:0.0:1789434551.078667:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434551.078672:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434551.078678:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434551.078690:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913838336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434551.078700:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434551.078703:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434551.078705:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434551.078709:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6920 00000100:00080000:0.0:1789434551.078712:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434551.078717:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434551.078721:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434551.078732:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913838464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434551.079257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434551.079276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434551.079285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434551/real 1789434551] req@ffff9811c8cd4000 x1876350913838336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434551.079292:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913838336/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434551.079297:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434551.079299:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434551.079301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838336:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434551.079305:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838464:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434551.079315:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434551.079320:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434551/real 1789434551] req@ffff9811c8cd6a00 x1876350913838464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434551.079326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913838464/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434551.079335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434551.079336:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434551.079338:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838464:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434551.974323:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434552.998177:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434554.022400:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434555.046464:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434556.071464:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434556.197970:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434556.197980:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434556.197984:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434556.197995:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6925 00000100:00080000:0.0:1789434556.198007:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434556.198019:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434556.198055:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913838592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434556.198086:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434556.198090:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434556.198092:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434556.198097:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6925 00000100:00080000:0.0:1789434556.198100:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434556.198106:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434556.198116:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434556.198129:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913838720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434556.198139:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434556.198142:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434556.198144:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434556.198147:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6925 00000100:00080000:0.0:1789434556.198150:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434556.198154:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434556.198159:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434556.198168:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913838848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434556.198178:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434556.198181:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434556.198183:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434556.198186:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6925 00000100:00080000:0.0:1789434556.198189:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434556.198193:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434556.198197:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434556.198207:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913838976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434556.198335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913838592:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434556.198406:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434556.198442:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434556/real 1789434556] req@ffff9811c8cd5180 x1876350913838592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434556.198460:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913838592/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434556.198470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434556.198474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434556.198478:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913838592:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434556.198485:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434556.198519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434556.198531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434556/real 1789434556] req@ffff9811c8cd7800 x1876350913838720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434556.198543:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913838720/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434556.198551:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434556.198554:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434556.198557:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434556.198564:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434556.198583:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434556.198594:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434556/real 1789434556] req@ffff9811c8cd7480 x1876350913838848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434556.198605:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913838848/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434556.198612:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434556.198615:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434556.198617:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434556.198622:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838976:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434556.198640:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434556.198650:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434556/real 1789434556] req@ffff9811c8cd7100 x1876350913838976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434556.198659:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913838976/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434556.198667:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434556.198670:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434556.198673:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913838976:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434557.094077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434558.118053:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434559.142184:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434560.166200:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434561.195203:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434561.318088:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434561.318097:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434561.318101:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434561.318110:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6930 00000100:00080000:2.0:1789434561.318130:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434561.318141:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434561.318519:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913839104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434561.318606:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913839104:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434561.318666:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434561.318698:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434561.318702:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434561.318703:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434561.318708:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6930 00000100:00080000:2.0:1789434561.318712:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434561.318717:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434561.318726:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434561.318738:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913839232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434561.319139:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434561/real 1789434561] req@ffff9811d2100e00 x1876350913839104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434561.319172:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913839104/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434561.319182:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434561.319186:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434561.319189:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913839104:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434561.319211:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434561.319235:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434561.319248:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434561/real 1789434561] req@ffff9811d2101f80 x1876350913839232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434561.319259:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913839232/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434561.319266:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434561.319269:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434561.319272:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839232:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434561.319773:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434561.319777:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434561.319780:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434561.319785:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6930 00000100:00080000:2.0:1789434561.319789:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434561.319796:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434561.319801:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434561.319816:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913839360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434561.319851:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434561.319855:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434561.319856:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434561.319861:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6930 00000100:00080000:2.0:1789434561.319864:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434561.319869:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434561.319873:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434561.319885:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913839488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434561.320236:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434561.320267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434561.320708:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434561/real 1789434561] req@ffff9811d2101180 x1876350913839360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434561.320728:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913839360/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434561.320742:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434561.320746:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434561.320749:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434561.320756:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839488:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434561.320784:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434561.320796:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434561/real 1789434561] req@ffff9811d2101f80 x1876350913839488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434561.320807:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913839488/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434561.320816:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434561.320819:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434561.320822:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839488:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434562.214057:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434563.237986:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434564.262066:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434565.286163:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434566.310595:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434566.438409:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434566.438419:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434566.438423:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434566.438433:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6936 00000100:00080000:0.0:1789434566.438442:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434566.438453:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434566.438488:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913839616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434566.438517:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434566.438521:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434566.438523:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434566.438527:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6936 00000100:00080000:0.0:1789434566.438531:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434566.438536:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434566.438545:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434566.438558:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913839744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434566.438568:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434566.438571:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434566.438572:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434566.438576:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6936 00000100:00080000:0.0:1789434566.438579:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434566.438583:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434566.438588:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434566.438597:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913839872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434566.438607:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434566.438611:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434566.438612:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434566.438615:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6936 00000100:00080000:0.0:1789434566.438619:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434566.438623:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434566.438627:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434566.438639:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913840000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434566.438825:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913839616:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434566.438936:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434566.438984:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434566/real 1789434566] req@ffff9811c8cd6680 x1876350913839616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434566.439004:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913839616/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434566.439016:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434566.439021:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434566.439025:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913839616:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434566.439033:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839744:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434566.439058:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434566.439071:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434566/real 1789434566] req@ffff9811c8cd4700 x1876350913839744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434566.439083:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913839744/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434566.439090:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434566.439094:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434566.439097:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434566.439103:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434566.439122:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434566.439134:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434566/real 1789434566] req@ffff9811c8cd5c00 x1876350913839872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434566.439146:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913839872/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434566.439154:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434566.439157:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434566.439161:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913839872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434566.439167:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840000:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434566.439187:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434566.439198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434566/real 1789434566] req@ffff9811c8cd7100 x1876350913840000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434566.439210:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913840000/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434566.439219:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434566.439222:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434566.439225:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840000:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434567.334020:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434568.358137:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434569.382073:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434570.406617:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434571.430023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434571.558021:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434571.558031:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434571.558036:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434571.558047:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6941 00000100:00080000:2.0:1789434571.558056:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434571.558067:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434571.558102:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913840128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434571.558133:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434571.558136:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434571.558138:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434571.558142:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6941 00000100:00080000:2.0:1789434571.558145:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434571.558150:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434571.558160:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434571.558173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913840256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434571.558182:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434571.558185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434571.558187:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434571.558190:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6941 00000100:00080000:2.0:1789434571.558193:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434571.558197:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434571.558210:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434571.558271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913840128:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434571.558329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434571.558365:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434571/real 1789434571] req@ffff9811d2100e00 x1876350913840128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434571.558382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913840128/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434571.558392:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434571.558395:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434571.558398:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913840128:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434571.558404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434571.558425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434571.558436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434571/real 1789434571] req@ffff9811d2101880 x1876350913840256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434571.558448:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913840256/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434571.558460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434571.558463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434571.558466:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789434571.558525:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913840384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434571.558547:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434571.558551:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434571.558552:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434571.558556:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6941 00000100:00080000:2.0:1789434571.558560:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434571.558565:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434571.558571:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434571.558581:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913840512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434571.558644:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434571.558665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434571.558677:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434571/real 1789434571] req@ffff9811d2101500 x1876350913840384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434571.558688:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913840384/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434571.558695:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434571.558698:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434571.558701:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434571.558707:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840512:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434571.558725:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434571.558735:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434571/real 1789434571] req@ffff9811d2102a00 x1876350913840512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434571.558746:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913840512/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434571.558753:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434571.558755:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434571.558759:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840512:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434572.454010:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434573.478236:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434574.503065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434575.525992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434576.550005:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434576.680344:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434576.680351:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434576.680355:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434576.680362:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6946 00000100:00080000:1.0:1789434576.680369:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434576.680378:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434576.680407:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913840640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434576.680436:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434576.680440:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434576.680442:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434576.680447:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6946 00000100:00080000:1.0:1789434576.680450:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434576.680455:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434576.680465:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434576.680477:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913840768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434576.680487:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434576.680490:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434576.680491:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434576.680495:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6946 00000100:00080000:1.0:1789434576.680498:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434576.680502:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434576.680507:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434576.680516:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913840896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434576.680526:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434576.680529:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434576.680531:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434576.680534:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6946 00000100:00080000:1.0:1789434576.680538:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434576.680542:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434576.680547:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434576.680556:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913841024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434576.681231:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913840640:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434576.681292:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434576.681329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434576/real 1789434576] req@ffff9811d0bdc000 x1876350913840640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434576.681347:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913840640/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434576.681357:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434576.681361:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434576.681365:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913840640:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434576.681371:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840768:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434576.681392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434576.681406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434576/real 1789434576] req@ffff9811d0bdd880 x1876350913840768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434576.681417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913840768/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434576.681425:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434576.681428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434576.681432:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434576.681437:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840896:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434576.681454:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434576.681721:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434576/real 1789434576] req@ffff9811d0bded80 x1876350913840896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434576.681741:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913840896/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434576.681748:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434576.681751:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434576.681755:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913840896:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434576.681760:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841024:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434576.681781:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434576.681792:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434576/real 1789434576] req@ffff9811d0bdfb80 x1876350913841024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434576.681803:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913841024/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434576.681811:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434576.681814:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434576.681817:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841024:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434577.574009:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434578.598087:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434579.622012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434580.646142:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434581.670043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434581.797992:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434581.798003:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434581.798021:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434581.798032:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6951 00000100:00080000:1.0:1789434581.798040:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434581.798052:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434581.798085:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913841152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434581.798114:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434581.798132:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434581.798135:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434581.798139:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6951 00000100:00080000:1.0:1789434581.798142:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434581.798147:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434581.798157:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434581.798170:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913841280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434581.798179:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913841152:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434581.798183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434581.798189:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434581.798191:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434581.798195:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6951 00000100:00080000:1.0:1789434581.798199:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434581.798203:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434581.798207:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434581.798217:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913841408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434581.798225:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434581.798228:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434581.798229:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434581.798232:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6951 00000100:00080000:1.0:1789434581.798236:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434581.798240:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434581.798245:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789434581.798250:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434581.798261:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913841536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434581.798292:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434581/real 1789434581] req@ffff9811d0bdf480 x1876350913841152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434581.798309:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913841152/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434581.798320:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434581.798323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434581.798327:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913841152:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434581.798347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434581.798367:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434581.798379:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434581/real 1789434581] req@ffff9811d0bde680 x1876350913841280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434581.798396:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913841280/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434581.798404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434581.798407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434581.798410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434581.798415:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434581.798433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434581.798443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434581/real 1789434581] req@ffff9811d0bdc380 x1876350913841408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434581.798453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913841408/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434581.798460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434581.798463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434581.798466:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434581.798472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841536:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434581.798488:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434581.798499:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434581/real 1789434581] req@ffff9811d0bdd180 x1876350913841536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434581.798508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913841536/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434581.798529:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434581.798532:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434581.798535:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841536:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434582.694372:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434583.718031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434584.742308:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434585.766089:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434586.790078:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434586.917939:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434586.917949:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434586.917953:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434586.917963:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6956 00000100:00080000:1.0:1789434586.917973:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434586.917984:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434586.918025:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913841664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434586.918115:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913841664:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789434586.918154:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434586.918158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434586.918160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434586.918165:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6956 00000100:00080000:1.0:1789434586.918168:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434586.918174:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434586.918181:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789434586.918183:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434586.918195:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913841792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434586.918204:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434586.918207:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434586.918209:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434586.918211:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6956 00000100:00080000:1.0:1789434586.918214:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789434586.918216:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434586/real 1789434586] req@ffff9811d0bdd500 x1876350913841664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434586.918219:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434586.918223:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434586.918231:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913841664/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434586.918233:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913841920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434586.918242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434586.918243:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434586.918246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:1.0:1789434586.918246:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1789434586.918249:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913841664:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434586.918271:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434586.918304:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434586.918313:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434586.918317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434586/real 1789434586] req@ffff9811d0bddc00 x1876350913841792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434586.918317:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6956 00000100:00080000:1.0:1789434586.918321:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434586.918326:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434586.918328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913841792/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789434586.918331:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789434586.918336:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434586.918339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434586.918342:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913842048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434586.918343:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434586.918348:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434586.918366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434586.918376:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434586/real 1789434586] req@ffff9811d0bdd180 x1876350913841920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434586.918387:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913841920/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434586.918393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434586.918395:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434586.918398:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913841920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434586.918411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842048:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434586.918430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434586.918441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434586/real 1789434586] req@ffff9811d0bdc380 x1876350913842048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434586.918458:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913842048/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434586.918464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434586.918465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434586.918468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842048:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434587.814141:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434588.838064:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434589.862086:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434590.886020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434591.910024:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434592.038105:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434592.038115:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434592.038119:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434592.038129:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6961 00000100:00080000:0.0:1789434592.038136:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434592.038147:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434592.038182:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913842176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434592.038213:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434592.038217:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434592.038218:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434592.038222:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6961 00000100:00080000:0.0:1789434592.038225:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434592.038230:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434592.038240:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434592.038252:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913842304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434592.038262:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434592.038264:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434592.038266:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434592.038270:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6961 00000100:00080000:0.0:1789434592.038273:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434592.038277:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434592.038281:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434592.038291:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913842432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434592.038301:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434592.038304:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434592.038305:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434592.038309:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6961 00000100:00080000:0.0:1789434592.038312:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434592.038316:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434592.038320:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434592.038330:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913842560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434592.038399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913842176:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434592.038487:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434592.038516:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434592.038539:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434592.934092:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434593.958012:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434594.982053:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789434595.110101:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:2.0:1789434595.110142:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789434595.110156:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434595.110269:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434592/real 1789434595] req@ffff9811c8cd7480 x1876350913842176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434595.110294:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913842176/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434595.110311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434595.110315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434595.110320:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913842176:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789434595.110373:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434595.110440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434592/real 1789434595] req@ffff9811c8cd7800 x1876350913842304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434595.110455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913842304/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434595.110463:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434595.110466:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434595.110469:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789434595.110510:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434595.110575:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434592/real 1789434595] req@ffff9811c8cd5180 x1876350913842432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434595.110590:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913842432/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434595.110598:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434595.110601:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434595.110605:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789434595.110644:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434595.111117:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434592/real 1789434595] req@ffff9811c8cd6a00 x1876350913842560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434595.111134:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913842560/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434617 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434595.111143:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434595.111147:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434595.111150:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842560:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789434596.006036:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434597.029996:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434597.157948:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434597.157958:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434597.157963:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434597.157974:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6966 00000100:00080000:0.0:1789434597.157984:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434597.157996:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434597.158053:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913842688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434597.158084:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434597.158088:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434597.158090:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434597.158094:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6966 00000100:00080000:0.0:1789434597.158097:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434597.158103:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434597.158112:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434597.158133:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913842816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434597.158142:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434597.158146:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434597.158148:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434597.158151:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6966 00000100:00080000:0.0:1789434597.158154:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434597.158157:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434597.158161:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434597.158169:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913842944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434597.158183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434597.158186:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434597.158187:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434597.158190:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6966 00000100:00080000:0.0:1789434597.158193:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434597.158197:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434597.158200:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434597.158208:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913843072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434597.158338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913842688:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434597.158395:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434597.158438:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434597/real 1789434597] req@ffff9811c8cd4000 x1876350913842688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434597.158454:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913842688/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434597.158462:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434597.158465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434597.158468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913842688:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434597.158475:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434597.158496:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434597.158507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434597/real 1789434597] req@ffff9811c8cd6300 x1876350913842816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434597.158517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913842816/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434597.158523:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434597.158526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434597.158528:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434597.158533:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434597.158550:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434597.158560:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434597/real 1789434597] req@ffff9811c8cd7b80 x1876350913842944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434597.158569:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913842944/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434597.158576:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434597.158578:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434597.158581:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913842944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434597.158586:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843072:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434597.158603:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434597.158612:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434597/real 1789434597] req@ffff9811c8cd4a80 x1876350913843072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434597.158622:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913843072/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434597.158629:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434597.158632:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434597.158635:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843072:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434598.054056:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434599.078083:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434600.102214:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434601.126106:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789434602.150060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434602.280061:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434602.280071:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434602.280074:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434602.280082:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6971 00000100:00080000:0.0:1789434602.280090:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434602.280099:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434602.280128:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913843200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434602.280151:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434602.280154:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434602.280156:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434602.280161:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6971 00000100:00080000:0.0:1789434602.280164:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434602.280168:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434602.280177:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434602.280189:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913843328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434602.280198:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434602.280201:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434602.280202:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434602.280207:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6971 00000100:00080000:0.0:1789434602.280210:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434602.280214:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434602.280218:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434602.280229:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913843456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434602.280238:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434602.280241:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434602.280243:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434602.280246:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6971 00000100:00080000:0.0:1789434602.280249:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434602.280253:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434602.280257:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434602.280265:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913843200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434602.280267:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913843584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434602.280323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434602.280357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434602/real 1789434602] req@ffff9811c8cd5500 x1876350913843200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434602.280372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913843200/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434602.280381:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434602.280384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434602.280387:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913843200:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434602.280394:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434602.280558:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434602.280570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434602/real 1789434602] req@ffff9811c8cd6d80 x1876350913843328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434602.280582:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913843328/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434602.280590:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434602.280593:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434602.280596:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434602.280601:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434602.280615:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434602.280623:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434602/real 1789434602] req@ffff9811c8cd5f80 x1876350913843456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434602.280633:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913843456/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434602.280646:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434602.280649:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434602.280652:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434602.280672:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843584:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434602.280686:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434602.280693:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434602/real 1789434602] req@ffff9811c8cd4a80 x1876350913843584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434602.280704:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913843584/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434602.280712:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434602.280714:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434602.280718:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843584:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434603.174030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434604.198069:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434605.225121:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434606.246062:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434607.270049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434607.398077:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434607.398087:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434607.398092:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434607.398262:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6976 00000100:00080000:0.0:1789434607.398272:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434607.398283:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434607.398317:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913843712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434607.398395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913843712:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434607.398474:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434607.398720:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434607.398725:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434607.398727:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434607.398732:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6976 00000100:00080000:0.0:1789434607.398736:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434607.398742:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434607.398750:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434607.398764:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913843840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434607.398996:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434607/real 1789434607] req@ffff9811c8cd7b80 x1876350913843712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434607.399015:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913843712/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434607.399024:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434607.399028:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434607.399032:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913843712:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434607.399054:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434607.399079:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434607.399091:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434607/real 1789434607] req@ffff9811c8cd6300 x1876350913843840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434607.399103:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913843840/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434607.399227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434607.399231:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434607.399236:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843840:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434607.399609:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434607.399614:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434607.399616:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434607.399620:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6976 00000100:00080000:0.0:1789434607.399625:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434607.399631:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434607.399638:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434607.399653:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913843968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434607.399665:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434607.399668:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434607.399670:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434607.399673:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6976 00000100:00080000:0.0:1789434607.399676:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434607.399681:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434607.399685:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434607.399766:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913844096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434607.399846:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434607.399876:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434607.399890:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434607/real 1789434607] req@ffff9811c8cd4000 x1876350913843968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434607.399901:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913843968/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434607.399909:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434607.399912:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434607.399915:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913843968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434607.399921:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844096:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434607.399951:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434607.399961:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434607/real 1789434607] req@ffff9811c8cd6a00 x1876350913844096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434607.399972:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913844096/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434607.399980:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434607.399983:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434607.399986:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844096:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434608.294593:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434609.318083:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434610.343989:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434611.366392:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434612.390001:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434612.518179:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434612.518189:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434612.518193:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434612.518204:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6982 00000100:00080000:1.0:1789434612.518215:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434612.518226:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434612.518262:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913844224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434612.518292:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434612.518295:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434612.518297:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434612.518301:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6982 00000100:00080000:1.0:1789434612.518305:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434612.518309:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434612.518318:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434612.518330:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913844352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434612.518346:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434612.518350:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434612.518351:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434612.518355:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6982 00000100:00080000:1.0:1789434612.518358:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434612.518363:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434612.518367:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434612.518377:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913844480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434612.518387:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434612.518390:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434612.518392:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434612.518395:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6982 00000100:00080000:1.0:1789434612.518399:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434612.518403:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434612.518408:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434612.518418:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913844608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434612.518493:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913844224:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434612.518559:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434612.518600:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434612/real 1789434612] req@ffff9811d0bde680 x1876350913844224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434612.518617:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913844224/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434612.518628:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434612.518632:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434612.518635:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913844224:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434612.518643:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434612.518665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434612.518675:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434612/real 1789434612] req@ffff9811d0bdf480 x1876350913844352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434612.518682:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913844352/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434612.518687:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434612.518689:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434612.518691:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434612.518695:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434612.518706:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434612.518711:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434612/real 1789434612] req@ffff9811d0bdfb80 x1876350913844480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434612.518722:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913844480/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434612.518739:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434612.518743:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434612.518746:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434612.518751:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844608:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434612.518769:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434612.518779:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434612/real 1789434612] req@ffff9811d0bded80 x1876350913844608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434612.518796:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913844608/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434612.518804:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434612.518806:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434612.518809:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844608:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434613.414042:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434614.438231:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434615.462132:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434616.486136:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434617.510016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434617.637948:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434617.637966:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434617.637974:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434617.637985:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6987 00000100:00080000:1.0:1789434617.637995:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434617.638007:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434617.638043:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913844736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434617.638139:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913844736:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434617.638206:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434617.638240:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434617.638244:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434617.638246:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434617.638250:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6987 00000100:00080000:1.0:1789434617.638254:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434617.638260:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434617.638269:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434617.638293:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913844864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434617.638562:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434617/real 1789434617] req@ffff9811d0bdd880 x1876350913844736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434617.638579:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913844736/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434617.638589:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434617.638592:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434617.638596:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913844736:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434617.638629:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434617.638654:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434617.638667:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434617/real 1789434617] req@ffff9811d0bdc000 x1876350913844864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434617.638678:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913844864/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434617.638692:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434617.638696:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434617.638699:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844864:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789434617.638869:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434617.638874:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434617.638876:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434617.638880:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6987 00000100:00080000:1.0:1789434617.638885:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434617.638890:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434617.638896:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434617.638908:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913844992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434617.638950:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434617.638954:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1789434617.638955:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844992:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789434617.638956:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434617.638964:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6987 00000100:00080000:1.0:1789434617.638968:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434617.638974:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434617.638976:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789434617.638980:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434617.638988:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434617/real 1789434617] req@ffff9811d0bdc700 x1876350913844992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434617.638991:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913845120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434617.639000:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913844992/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434617.639007:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434617.639010:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434617.639013:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913844992:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434617.639024:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845120:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434617.639043:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434617.639051:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434617/real 1789434617] req@ffff9811d0bdea00 x1876350913845120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434617.639058:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913845120/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434617.639063:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434617.639065:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434617.639067:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845120:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434618.534056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434619.558029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434620.582592:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434621.606048:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434622.630245:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434622.758035:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434622.758043:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434622.758046:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434622.758055:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6992 00000100:00080000:1.0:1789434622.758062:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434622.758071:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434622.758102:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913845248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434622.758127:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434622.758130:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434622.758137:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434622.758141:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6992 00000100:00080000:1.0:1789434622.758145:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434622.758149:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434622.758164:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434622.758178:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913845376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434622.758188:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434622.758191:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434622.758193:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434622.758196:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6992 00000100:00080000:1.0:1789434622.758198:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434622.758202:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434622.758205:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434622.758212:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913845504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434622.758219:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434622.758221:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434622.758221:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434622.758223:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6992 00000100:00080000:1.0:1789434622.758225:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434622.758227:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434622.758233:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434622.758239:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913845632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434622.758301:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913845248:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434622.758348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434622.758375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434622/real 1789434622] req@ffff9811d0bdf800 x1876350913845248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434622.758386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913845248/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434622.758393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434622.758396:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434622.758398:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913845248:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434622.758402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434622.758414:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434622.758421:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434622/real 1789434622] req@ffff9811d0bde300 x1876350913845376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434622.758428:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913845376/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434622.758432:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434622.758434:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434622.758436:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434622.758439:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434622.758450:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434622.758460:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434622/real 1789434622] req@ffff9811d0bdf100 x1876350913845504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434622.758469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913845504/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434622.758476:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434622.758479:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434622.758482:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434622.758487:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845632:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434622.758505:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434622.758514:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434622/real 1789434622] req@ffff9811d0bdea00 x1876350913845632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434622.758521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913845632/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434622.758528:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434622.758531:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434622.758534:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845632:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434623.654275:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434624.678028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434625.702148:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434626.728294:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434627.750696:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434627.878055:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434627.878065:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434627.878069:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434627.878084:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 6997 00000100:00080000:3.0:1789434627.878095:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434627.878107:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434627.878142:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913845760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434627.878166:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434627.878170:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434627.878172:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434627.878176:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6997 00000100:00080000:3.0:1789434627.878180:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434627.878187:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434627.878200:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434627.878203:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913845760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434627.878218:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913845888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434627.878229:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434627.878233:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434627.878235:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434627.878239:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6997 00000100:00080000:3.0:1789434627.878243:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434627.878253:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434627.878258:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789434627.878261:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789434627.878273:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913846016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434627.878284:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434627.878287:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434627.878289:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434627.878293:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 6997 00000100:00080000:3.0:1789434627.878297:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789434627.878303:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434627/real 1789434627] req@ffff9811d1bc7100 x1876350913845760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789434627.878305:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434627.878313:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434627.878326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913845760/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434627.878340:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789434627.878341:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913846144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434627.878347:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434627.878354:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913845760:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434627.878379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434627.878412:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434627.878423:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434627/real 1789434627] req@ffff9811d1bc4700 x1876350913845888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434627.878434:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913845888/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434627.878441:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434627.878444:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434627.878447:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913845888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434627.878452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434627.878469:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434627.878705:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434627/real 1789434627] req@ffff9811d1bc6300 x1876350913846016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434627.878718:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913846016/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434627.878726:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434627.878729:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434627.878733:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434627.878739:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846144:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434627.878762:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434627.878772:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434627/real 1789434627] req@ffff9811d1bc5500 x1876350913846144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434627.878798:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913846144/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434627.878806:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434627.878809:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434627.878812:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846144:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434628.774023:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434629.797973:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434630.822151:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434631.846338:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434632.870066:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434632.997951:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434632.997961:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434632.997965:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434632.997976:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7002 00000100:00080000:3.0:1789434632.997984:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434632.997996:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434632.998031:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913846272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434632.998058:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434632.998062:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434632.998064:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434632.998068:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7002 00000100:00080000:3.0:1789434632.998071:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434632.998075:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434632.998084:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434632.998098:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913846400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434632.998108:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434632.998111:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434632.998113:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434632.998117:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7002 00000100:00100000:0.0:1789434632.998122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913846272:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434632.998124:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434632.998133:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434632.998138:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434632.998153:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913846528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434632.998185:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434632.998222:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434632/real 1789434632] req@ffff9811d1bc5f80 x1876350913846272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434632.998248:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913846272/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434632.998258:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434632.998263:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434632.998266:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913846272:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434632.998274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846400:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434632.998295:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434632.998307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434632/real 1789434632] req@ffff9811d1bc4000 x1876350913846400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434632.998319:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913846400/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434632.998326:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434632.998330:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434632.998333:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846400:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434632.998626:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434632.998629:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434632.998631:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434632.998636:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7002 00000100:00080000:3.0:1789434632.998640:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434632.998645:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434632.998651:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434632.998665:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913846656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434632.998719:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434632.998755:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434632.998768:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434632/real 1789434632] req@ffff9811d1bc5c00 x1876350913846528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434632.998779:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913846528/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434632.998788:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434632.998791:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434632.998794:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434632.998800:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846656:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434632.998820:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434632.998848:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434632/real 1789434632] req@ffff9811d1bc6680 x1876350913846656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434632.998862:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913846656/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434632.998876:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434632.998879:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434632.998883:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846656:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434633.896459:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434634.918342:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434635.944218:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434636.966518:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434637.990003:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434638.117953:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434638.117963:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434638.117967:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434638.117976:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7007 00000100:00080000:1.0:1789434638.117984:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434638.117996:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434638.118039:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913846784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434638.118066:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434638.118069:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434638.118071:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434638.118076:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7007 00000100:00080000:1.0:1789434638.118079:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434638.118085:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434638.118095:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434638.118102:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913846784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434638.118109:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913846912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434638.118120:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434638.118124:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434638.118125:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434638.118129:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7007 00000100:00080000:1.0:1789434638.118132:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434638.118137:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434638.118141:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434638.118152:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913847040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434638.118162:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434638.118162:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434638.118168:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434638.118170:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434638.118175:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7007 00000100:00080000:1.0:1789434638.118179:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434638.118185:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434638.118190:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434638.118202:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913847168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434638.118213:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434638/real 1789434638] req@ffff9811d0bdc700 x1876350913846784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434638.118232:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913846784/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434638.118242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434638.118247:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434638.118251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913846784:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434638.118274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434638.118296:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434638.118307:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434638/real 1789434638] req@ffff9811d0bdc000 x1876350913846912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434638.118318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913846912/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434638.118326:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434638.118329:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434638.118331:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913846912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434638.118335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434638.118348:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434638.118355:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434638/real 1789434638] req@ffff9811d0bdd880 x1876350913847040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434638.118372:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913847040/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434638.118380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434638.118383:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434638.118386:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434638.118392:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847168:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434638.118411:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434638.118421:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434638/real 1789434638] req@ffff9811d0bded80 x1876350913847168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434638.118431:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913847168/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434638.118439:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434638.118442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434638.118445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847168:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434639.014054:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434640.038557:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434641.062050:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434642.086240:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434643.111099:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434643.238284:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434643.238302:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434643.238305:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434643.238321:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7012 00000100:00080000:1.0:1789434643.238330:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434643.238356:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434643.238418:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913847296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434643.238460:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434643.238464:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434643.238466:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434643.238470:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7012 00000100:00080000:1.0:1789434643.238474:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434643.238479:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434643.238488:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434643.238518:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913847424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434643.238528:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434643.238531:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434643.238533:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434643.238536:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7012 00000100:00080000:1.0:1789434643.238540:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434643.238544:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434643.238564:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434643.238575:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913847552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434643.238583:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434643.238586:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434643.238588:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434643.238591:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7012 00000100:00080000:1.0:1789434643.238594:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434643.238598:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434643.238619:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434643.238630:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913847680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434643.238719:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913847296:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434643.238787:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434643.238853:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434643/real 1789434643] req@ffff9811d0bdfb80 x1876350913847296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434643.238876:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913847296/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434643.238890:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434643.238894:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434643.238898:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913847296:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434643.238911:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434643.238949:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434643.238962:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434643/real 1789434643] req@ffff9811d0bdf480 x1876350913847424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434643.238974:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913847424/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434643.238999:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434643.239002:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434643.239006:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434643.239011:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847552:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434643.239047:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434643.239069:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434643/real 1789434643] req@ffff9811d0bde680 x1876350913847552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434643.239090:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913847552/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434643.239104:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434643.239107:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434643.239110:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434643.239122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847680:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434643.239150:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434643.239167:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434643/real 1789434643] req@ffff9811d0bdc380 x1876350913847680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434643.239177:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913847680/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434643.239189:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434643.239192:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434643.239195:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847680:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434644.134049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434645.157971:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434646.182981:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434647.206731:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434648.230182:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434648.359751:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434648.359765:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434648.359771:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434648.359783:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7017 00000100:00080000:2.0:1789434648.359795:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434648.359808:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434648.359888:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913847808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434648.359923:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434648.359927:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434648.359929:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434648.359934:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7017 00000100:00080000:2.0:1789434648.359937:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434648.359943:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434648.359965:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434648.359980:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913847936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434648.359993:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434648.359997:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434648.360000:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789434648.360009:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913847808:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434648.360010:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7017 00000100:00080000:2.0:1789434648.360016:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434648.360035:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434648.360041:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434648.360055:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913848064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434648.360072:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434648.360076:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434648.360078:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434648.360083:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7017 00000100:00080000:2.0:1789434648.360087:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789434648.360095:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434648.360095:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434648.360106:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434648.360123:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913848192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434648.360158:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434648/real 1789434648] req@ffff9811d2103b80 x1876350913847808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434648.360184:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913847808/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434648.360196:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434648.360201:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434648.360205:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913847808:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434648.360230:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434648.360255:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434648.360268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434648/real 1789434648] req@ffff9811d2101c00 x1876350913847936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434648.360281:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913847936/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434648.360289:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434648.360292:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434648.360296:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913847936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434648.360302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434648.360321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434648.360332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434648/real 1789434648] req@ffff9811d2102d80 x1876350913848064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434648.360343:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913848064/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434648.360350:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434648.360353:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434648.360356:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434648.360362:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848192:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434648.360382:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434648.360393:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434648/real 1789434648] req@ffff9811d2103800 x1876350913848192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434648.360406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913848192/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434648.360413:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434648.360416:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434648.360419:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848192:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434649.255472:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434650.278045:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434651.302004:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434652.327282:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434653.350064:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434653.478102:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434653.478113:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434653.478116:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434653.478126:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7023 00000100:00080000:1.0:1789434653.478138:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434653.478149:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434653.478185:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913848320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434653.478216:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434653.478220:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434653.478222:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434653.478226:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7023 00000100:00080000:1.0:1789434653.478230:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434653.478234:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434653.478245:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434653.478266:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913848448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434653.478277:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434653.478280:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434653.478282:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434653.478286:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7023 00000100:00080000:1.0:1789434653.478289:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434653.478293:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434653.478298:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434653.478309:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913848576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434653.478320:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434653.478324:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434653.478325:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434653.478328:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7023 00000100:00080000:1.0:1789434653.478332:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434653.478336:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434653.478341:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434653.478352:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913848704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434653.478443:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913848320:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434653.478498:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434653.478533:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434653/real 1789434653] req@ffff9811d0bdd180 x1876350913848320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434653.478550:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913848320/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434653.478560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434653.478564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434653.478567:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913848320:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434653.478575:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434653.479107:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434653.479124:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434653/real 1789434653] req@ffff9811d0bddc00 x1876350913848448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434653.479138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913848448/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434653.479153:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434653.479157:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434653.479160:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434653.479166:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434653.479189:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434653.479200:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434653/real 1789434653] req@ffff9811d0bdd500 x1876350913848576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434653.479209:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913848576/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434653.479216:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434653.479218:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434653.479221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434653.479226:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848704:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434653.479244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434653.479253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434653/real 1789434653] req@ffff9811d0bdc380 x1876350913848704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434653.479263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913848704/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434653.479269:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434653.479271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434653.479274:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848704:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434654.374285:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434655.398048:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434656.422182:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434657.446591:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434658.470288:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434658.597950:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434658.597959:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434658.597963:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434658.597973:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7028 00000100:00080000:1.0:1789434658.597982:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434658.597995:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434658.598032:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913848832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434658.598065:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434658.598069:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434658.598071:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434658.598075:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7028 00000100:00080000:1.0:1789434658.598079:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434658.598084:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434658.598094:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434658.598106:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913848960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434658.598115:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434658.598118:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434658.598120:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434658.598124:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7028 00000100:00080000:1.0:1789434658.598127:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434658.598131:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434658.598136:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434658.598145:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913849088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434658.598153:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434658.598156:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434658.598158:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434658.598161:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7028 00000100:00080000:1.0:1789434658.598165:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434658.598168:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434658.598173:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434658.598182:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913849216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434658.598315:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913848832:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434658.598648:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434658.598680:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434658.598702:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434659.494753:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434660.519621:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434661.542647:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434661.678034:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434661.678084:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434661.678296:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434661.678479:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434658/real 1789434661] req@ffff9811d0bde680 x1876350913848832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434661.678513:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913848832/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434661.678537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434661.678542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434661.678546:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913848832:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434661.678778:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434661.678925:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434658/real 1789434661] req@ffff9811d0bdf480 x1876350913848960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434661.678942:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913848960/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434661.678951:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434661.678955:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434661.678958:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913848960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434661.679301:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434661.680761:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434658/real 1789434661] req@ffff9811d0bdfb80 x1876350913849088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434661.680786:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913849088/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434661.680797:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434661.680801:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434661.680804:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434661.681626:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434661.683208:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434658/real 1789434661] req@ffff9811d0bded80 x1876350913849216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434683 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434661.683877:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913849216/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434661.683892:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434661.683897:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434661.683902:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849216:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434662.566191:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434663.590110:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434663.718038:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434663.718048:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434663.718051:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434663.718061:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7033 00000100:00080000:1.0:1789434663.718068:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434663.718079:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434663.718108:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913849344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434663.718138:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434663.718142:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434663.718143:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434663.718147:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7033 00000100:00080000:1.0:1789434663.718150:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434663.718154:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434663.718163:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434663.718174:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913849472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434663.718183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434663.718186:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434663.718188:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434663.719340:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7033 00000100:00080000:1.0:1789434663.719348:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434663.719357:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434663.719366:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434663.719390:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913849600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434663.719404:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434663.719407:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434663.719409:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434663.719412:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7033 00000100:00080000:1.0:1789434663.719414:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434663.719419:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434663.719423:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434663.719431:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913849728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434663.720454:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913849344:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434663.720539:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434663.720579:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434663/real 1789434663] req@ffff9811d0bdfb80 x1876350913849344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434663.720596:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913849344/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434663.720606:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434663.720610:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434663.720613:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913849344:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434663.720621:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434663.720656:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434663.720669:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434663/real 1789434663] req@ffff9811d0bdf480 x1876350913849472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434663.720680:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913849472/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434663.720687:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434663.720690:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434663.720693:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434663.720699:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434663.720717:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434663.720727:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434663/real 1789434663] req@ffff9811d0bde680 x1876350913849600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434663.720737:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913849600/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434663.720744:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434663.720748:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434663.720751:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849600:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434663.720757:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849728:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434663.720913:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434663.720929:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434663/real 1789434663] req@ffff9811d0bdd880 x1876350913849728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434663.720941:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913849728/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434663.720949:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434663.720951:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434663.720955:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849728:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434664.614604:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434665.638070:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434666.662061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434667.686631:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434668.710200:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434668.838071:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434668.838085:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434668.838091:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434668.838103:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7038 00000100:00080000:1.0:1789434668.838116:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434668.838131:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434668.838178:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913849856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434668.838219:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434668.838224:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434668.838227:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434668.838233:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7038 00000100:00080000:1.0:1789434668.838237:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434668.838243:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434668.838255:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434668.838270:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913849984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434668.838280:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434668.838283:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434668.838285:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434668.838288:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7038 00000100:00080000:1.0:1789434668.838292:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434668.838297:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434668.838301:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434668.838310:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913850112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434668.838320:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434668.838324:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434668.838326:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434668.838329:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7038 00000100:00080000:1.0:1789434668.838333:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434668.838337:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434668.838343:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434668.838349:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913849856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434668.838357:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913850240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434668.838431:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434668.838490:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434668/real 1789434668] req@ffff9811d0bdc000 x1876350913849856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434668.838509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913849856/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434668.838519:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434668.838523:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434668.838527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913849856:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434668.838534:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434668.838568:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434668.838581:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434668/real 1789434668] req@ffff9811d0bdc700 x1876350913849984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434668.838592:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913849984/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434668.838606:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434668.838610:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434668.838613:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913849984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434668.838620:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434668.838640:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434668.838652:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434668/real 1789434668] req@ffff9811d0bdea00 x1876350913850112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434668.838664:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913850112/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434668.838673:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434668.838676:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434668.838679:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434668.838707:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850240:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434668.838729:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434668.838740:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434668/real 1789434668] req@ffff9811d0bdf100 x1876350913850240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434668.838750:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913850240/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434668.838759:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434668.838761:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434668.838765:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850240:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434669.734261:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434670.758274:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434671.782077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434672.806171:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434673.830058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434673.957968:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434673.957981:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434673.957985:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434673.957998:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7043 00000100:00080000:2.0:1789434673.958009:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434673.958022:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434673.958061:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913850368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434673.958094:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434673.958098:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434673.958100:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434673.958104:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7043 00000100:00080000:2.0:1789434673.958108:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434673.958112:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434673.958121:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434673.958135:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913850496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434673.958145:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434673.958148:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434673.958150:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434673.958154:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7043 00000100:00080000:2.0:1789434673.958157:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434673.958161:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434673.958165:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434673.958175:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913850624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434673.958185:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434673.958189:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434673.958190:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434673.958194:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7043 00000100:00080000:2.0:1789434673.958197:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434673.958201:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434673.958206:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434673.958215:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913850752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434673.958289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913850368:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434673.958346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434673.958383:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434673/real 1789434673] req@ffff9811d2100000 x1876350913850368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434673.958400:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913850368/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434673.958410:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434673.958414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434673.958418:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913850368:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434673.958424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434673.958457:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434673.958472:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434673/real 1789434673] req@ffff9811d2103800 x1876350913850496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434673.958484:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913850496/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434673.958492:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434673.958494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434673.958498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434673.958503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434673.958523:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434673.958534:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434673/real 1789434673] req@ffff9811d2102d80 x1876350913850624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434673.958545:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913850624/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434673.958564:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434673.958568:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434673.958571:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434673.958577:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850752:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434673.958596:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434673.958607:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434673/real 1789434673] req@ffff9811d2101c00 x1876350913850752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434673.958618:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913850752/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434673.958626:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434673.958629:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434673.958632:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913850752:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434674.854240:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434675.878088:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434676.902233:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434677.925991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434678.950026:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434679.077943:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434679.077953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434679.077957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434679.077967:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7048 00000100:00080000:2.0:1789434679.077988:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434679.078001:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434679.078039:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913850880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434679.078068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434679.078071:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434679.078074:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434679.078078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7048 00000100:00080000:2.0:1789434679.078082:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434679.078086:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434679.078096:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434679.078109:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913851008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434679.078118:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434679.078121:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434679.078123:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434679.078140:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7048 00000100:00080000:2.0:1789434679.078143:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434679.078147:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434679.078152:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434679.078163:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913851136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434679.078173:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434679.078176:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434679.078178:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434679.078182:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7048 00000100:00080000:2.0:1789434679.078186:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434679.078191:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434679.078196:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434679.078206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913850880:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434679.078209:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913851264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434679.078270:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434679.078305:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434679/real 1789434679] req@ffff9811d2103b80 x1876350913850880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434679.078323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913850880/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434679.078333:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434679.078337:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434679.078341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913850880:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434679.078348:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434679.078369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434679.078381:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434679/real 1789434679] req@ffff9811d2102a00 x1876350913851008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434679.078393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913851008/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434679.078400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434679.078419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434679.078423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434679.078428:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434679.078447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434679.078457:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434679/real 1789434679] req@ffff9811d2101500 x1876350913851136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434679.078468:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913851136/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434679.078474:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434679.078477:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434679.078479:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434679.078501:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851264:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434679.078521:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434679.078531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434679/real 1789434679] req@ffff9811d2101880 x1876350913851264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434679.078542:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913851264/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434679.078549:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434679.078552:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434679.078555:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851264:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434679.974184:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434680.999087:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434682.022084:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434683.046631:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434684.070888:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434684.197951:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434684.197961:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434684.197964:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434684.197974:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7053 00000100:00080000:2.0:1789434684.197983:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434684.197994:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434684.198028:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913851392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434684.198100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913851392:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434684.198153:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434684.198181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434684.198185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434684.198187:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434684.198192:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7053 00000100:00080000:2.0:1789434684.198195:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434684.198200:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434684.198209:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434684.198220:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913851520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434684.198864:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434684/real 1789434684] req@ffff9811d2100e00 x1876350913851392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434684.198879:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913851392/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434684.198886:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434684.198889:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434684.198892:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913851392:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434684.198910:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434684.198928:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434684.198936:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434684/real 1789434684] req@ffff9811d2101f80 x1876350913851520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434684.198944:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913851520/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434684.198950:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434684.198952:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434684.198954:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851520:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434684.199347:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434684.199351:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434684.199354:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434684.199359:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7053 00000100:00080000:2.0:1789434684.199462:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434684.199470:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434684.199477:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434684.199582:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913851648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434684.199734:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851648:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434684.199763:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434684.200815:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434684.200821:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434684.200824:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434684.200830:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7053 00000100:00080000:2.0:1789434684.200849:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434684.200856:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434684.200863:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434684.200881:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913851776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434684.201685:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434684/real 1789434684] req@ffff9811d2101180 x1876350913851648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434684.201707:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913851648/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434684.201719:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434684.201724:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434684.201727:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434684.201754:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851776:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434684.201794:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434684.201809:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434684/real 1789434684] req@ffff9811d2101f80 x1876350913851776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434684.201821:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913851776/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434684.202476:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434684.202480:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434684.202482:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913851776:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434685.098455:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434686.118721:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434687.142038:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434688.166218:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434689.190030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434689.318378:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434689.318388:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434689.318393:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434689.318403:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7058 00000100:00080000:0.0:1789434689.318412:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434689.318422:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434689.318458:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913851904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434689.318492:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434689.318496:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434689.318497:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434689.318501:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7058 00000100:00080000:0.0:1789434689.318505:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434689.318509:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434689.318518:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434689.318529:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913852032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434689.318539:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434689.318542:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434689.318544:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434689.318548:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7058 00000100:00080000:0.0:1789434689.318551:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434689.318555:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434689.318559:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434689.318568:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913852160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434689.318579:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434689.318583:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434689.318585:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434689.318588:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7058 00000100:00080000:0.0:1789434689.318591:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434689.318595:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434689.318601:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434689.318601:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913851904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434689.318615:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913852288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434689.318665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434689.318705:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434689/real 1789434689] req@ffff9811c8cd5180 x1876350913851904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434689.318723:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913851904/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434689.318734:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434689.318738:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434689.318742:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913851904:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434689.318748:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434689.318769:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434689.318781:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434689/real 1789434689] req@ffff9811c8cd7800 x1876350913852032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434689.318792:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913852032/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434689.318799:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434689.318802:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434689.318805:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434689.318811:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434689.318883:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434689.318897:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434689/real 1789434689] req@ffff9811c8cd7480 x1876350913852160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434689.318908:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913852160/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434689.318915:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434689.318918:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434689.318921:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434689.318943:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852288:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434689.318960:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434689.318968:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434689/real 1789434689] req@ffff9811c8cd7100 x1876350913852288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434689.318977:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913852288/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434689.318984:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434689.318987:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434689.318990:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852288:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434690.214098:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434691.238061:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434692.262221:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434693.286040:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434694.310053:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434694.438002:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434694.438010:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434694.438015:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434694.438024:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7064 00000100:00080000:2.0:1789434694.438032:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434694.438042:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434694.438085:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913852416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434694.438113:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434694.438117:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434694.438119:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434694.438123:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7064 00000100:00080000:2.0:1789434694.438126:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434694.438136:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434694.438145:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434694.438169:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913852544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434694.438179:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434694.438182:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434694.438183:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434694.438186:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7064 00000100:00080000:2.0:1789434694.438190:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434694.438194:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434694.438199:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434694.438209:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913852672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434694.438224:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434694.438226:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434694.438228:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434694.438232:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7064 00000100:00080000:2.0:1789434694.438235:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434694.438239:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434694.438244:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434694.438253:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913852800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434694.438380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913852416:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434694.438438:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434694.438486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434694/real 1789434694] req@ffff9811d2100e00 x1876350913852416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434694.438512:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913852416/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434694.438522:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434694.438526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434694.438529:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913852416:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434694.438536:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434694.438557:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434694.438568:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434694/real 1789434694] req@ffff9811d2101880 x1876350913852544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434694.438580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913852544/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434694.438588:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434694.438590:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434694.438593:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434694.438598:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434694.438615:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434694.438626:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434694/real 1789434694] req@ffff9811d2101500 x1876350913852672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434694.438637:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913852672/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434694.438649:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434694.438652:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434694.438656:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434694.438662:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852800:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434694.438681:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434694.438690:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434694/real 1789434694] req@ffff9811d2102a00 x1876350913852800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434694.438701:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913852800/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434694.438710:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434694.438712:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434694.438716:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913852800:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434695.334047:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434696.359019:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434697.382416:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434698.406635:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434699.430156:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434699.557944:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434699.557954:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434699.557958:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434699.557968:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7069 00000100:00080000:2.0:1789434699.557977:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434699.557988:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434699.558024:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913852928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434699.558056:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434699.558060:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434699.558062:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434699.558068:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7069 00000100:00080000:2.0:1789434699.558072:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434699.558077:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434699.558104:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434699.558110:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913852928:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434699.558121:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913853056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434699.558133:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434699.558137:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434699.558139:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434699.558143:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7069 00000100:00080000:2.0:1789434699.558148:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434699.558153:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434699.558158:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789434699.558168:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789434699.558170:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913853184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434699.558204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434699/real 1789434699] req@ffff9811d2103b80 x1876350913852928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434699.558220:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913852928/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434699.558229:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434699.558232:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434699.558235:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913852928:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434699.558265:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853056:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434699.558287:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434699.558299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434699/real 1789434699] req@ffff9811d2101c00 x1876350913853056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434699.558311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913853056/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434699.558319:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434699.558323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434699.558326:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853056:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434699.558917:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434699.558922:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434699.558924:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434699.558928:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7069 00000100:00080000:2.0:1789434699.558932:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434699.558938:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434699.558943:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434699.558957:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913853312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434699.560661:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434699.560719:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434699.560741:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434699/real 1789434699] req@ffff9811d2102d80 x1876350913853184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434699.560755:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913853184/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434699.560764:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434699.560768:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434699.560771:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853184:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434699.560882:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853312:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434699.560906:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434699.560926:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434699/real 1789434699] req@ffff9811d2103800 x1876350913853312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434699.560939:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913853312/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434699.560948:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434699.560951:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434699.560954:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853312:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434700.454080:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434701.478003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434702.501952:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434703.526248:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434704.550007:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434704.678583:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434704.678592:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434704.678597:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434704.678606:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7074 00000100:00080000:1.0:1789434704.678616:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434704.678627:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434704.678662:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913853440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434704.678698:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434704.678702:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434704.678703:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434704.678707:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7074 00000100:00080000:1.0:1789434704.678711:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434704.678715:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434704.678725:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434704.678737:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913853568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434704.678751:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434704.678754:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434704.678756:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434704.678759:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7074 00000100:00080000:1.0:1789434704.678762:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434704.678766:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434704.678770:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434704.678781:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913853696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434704.678791:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434704.678794:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434704.678795:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434704.678799:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7074 00000100:00080000:1.0:1789434704.678802:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434704.678806:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434704.678810:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434704.678819:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913853824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434704.679418:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913853440:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434704.679478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434704.679516:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434704/real 1789434704] req@ffff9811d0bde300 x1876350913853440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434704.679534:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913853440/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434704.679544:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434704.679548:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434704.679552:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913853440:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434704.679559:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434704.679580:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434704.679592:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434704/real 1789434704] req@ffff9811d0bdf800 x1876350913853568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434704.679603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913853568/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434704.679611:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434704.679614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434704.679617:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434704.679624:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434704.679641:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434704.679650:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434704/real 1789434704] req@ffff9811d0bdf100 x1876350913853696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434704.679660:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913853696/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434704.679667:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434704.679670:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434704.679672:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434704.679677:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853824:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434704.679693:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434704.679701:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434704/real 1789434704] req@ffff9811d0bdea00 x1876350913853824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434704.679711:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913853824/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434704.679718:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434704.679720:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434704.679723:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913853824:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434705.574252:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434706.598063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434707.622241:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434708.646008:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434709.670005:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434709.798204:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434709.798214:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434709.798218:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434709.798228:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7079 00000100:00080000:2.0:1789434709.798237:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434709.798248:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434709.798294:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913853952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434709.798361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913853952:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434709.798422:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434709.798454:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434709.798458:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434709.798460:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434709.798465:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7079 00000100:00080000:2.0:1789434709.798469:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434709.798474:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434709.798483:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434709.798498:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913854080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434709.799023:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434709/real 1789434709] req@ffff9811d2100000 x1876350913853952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434709.799043:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913853952/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434709.799052:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434709.799056:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434709.799060:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913853952:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434709.799082:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434709.799107:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434709.799120:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434709/real 1789434709] req@ffff9811d2103800 x1876350913854080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434709.799138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913854080/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434709.799145:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434709.799148:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434709.799151:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854080:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434709.799390:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434709.799395:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434709.799397:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434709.799401:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7079 00000100:00080000:2.0:1789434709.799405:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434709.799410:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434709.799417:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434709.799430:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913854208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434709.799472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854208:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434709.799493:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434709.799513:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434709.799516:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434709.799517:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434709.799521:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7079 00000100:00080000:2.0:1789434709.799524:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434709.799529:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434709.799533:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434709.799544:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913854336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434709.799575:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434709/real 1789434709] req@ffff9811d2102d80 x1876350913854208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434709.799586:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913854208/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434709.799592:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434709.799596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434709.799598:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854208:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434709.799608:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854336:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434709.799625:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434709.799634:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434709/real 1789434709] req@ffff9811d2101c00 x1876350913854336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434709.799643:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913854336/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434709.799655:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434709.799658:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434709.799661:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854336:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434710.694224:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434711.721820:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434712.743208:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434713.767501:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434714.790147:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434714.918306:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434714.918316:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434714.918320:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434714.918330:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7084 00000100:00080000:2.0:1789434714.918339:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434714.918416:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434714.918453:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913854464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434714.918554:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434714.918559:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434714.918561:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434714.918566:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7084 00000100:00080000:2.0:1789434714.918570:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434714.918576:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434714.918586:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434714.918600:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913854592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434714.918610:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434714.918613:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434714.918615:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434714.918619:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7084 00000100:00080000:2.0:1789434714.918622:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434714.918627:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434714.918631:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434714.918641:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913854720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434714.918652:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434714.918655:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434714.918657:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434714.918661:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7084 00000100:00080000:2.0:1789434714.918664:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434714.918669:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434714.918674:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434714.918685:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913854848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434714.918698:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913854464:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434714.918757:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434714.918796:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434714/real 1789434714] req@ffff9811d2103b80 x1876350913854464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434714.918816:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913854464/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434714.918826:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434714.918859:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434714.918864:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913854464:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434714.918871:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434714.918956:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434714.919252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434714/real 1789434714] req@ffff9811d2102a00 x1876350913854592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434714.919266:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913854592/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434714.919274:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434714.919277:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434714.919281:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434714.919287:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854720:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434714.919309:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434714.919320:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434714/real 1789434714] req@ffff9811d2101500 x1876350913854720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434714.919332:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913854720/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434714.919340:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434714.919343:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434714.919346:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854720:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434714.919369:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854848:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434714.919387:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434714.919397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434714/real 1789434714] req@ffff9811d2101880 x1876350913854848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434714.919408:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913854848/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434714.919416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434714.919420:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434714.919423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913854848:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434715.814073:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434716.838254:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434717.862022:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434718.886295:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434719.910541:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434720.039515:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434720.039525:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434720.039529:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434720.039540:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7089 00000100:00080000:0.0:1789434720.039548:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434720.039559:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434720.039592:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913854976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434720.039625:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434720.039629:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434720.039630:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434720.039634:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7089 00000100:00080000:0.0:1789434720.039638:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434720.039642:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434720.039651:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434720.039665:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913855104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434720.039674:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434720.039677:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434720.039679:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434720.039682:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7089 00000100:00080000:0.0:1789434720.039685:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434720.039690:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434720.039694:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434720.039703:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913855232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434720.039719:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434720.039722:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434720.039724:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434720.039727:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7089 00000100:00080000:0.0:1789434720.039730:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434720.039734:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434720.039739:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434720.039750:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913855360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434720.039854:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913854976:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434720.039911:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434720.040049:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434720/real 1789434720] req@ffff9811c8cd5c00 x1876350913854976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434720.040074:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913854976/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434720.040083:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434720.040087:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434720.040091:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913854976:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434720.040106:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434720.040130:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434720.040143:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434720/real 1789434720] req@ffff9811c8cd4700 x1876350913855104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434720.040154:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913855104/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434720.040162:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434720.040165:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434720.040168:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434720.040174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855232:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434720.040194:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434720.040204:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434720/real 1789434720] req@ffff9811c8cd6680 x1876350913855232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434720.040214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913855232/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434720.040227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434720.040230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434720.040233:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855232:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434720.040239:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855360:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434720.040257:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434720.040268:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434720/real 1789434720] req@ffff9811c8cd7100 x1876350913855360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434720.040279:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913855360/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434720.040286:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434720.040290:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434720.040293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855360:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434720.934964:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434721.958679:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434722.983070:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434724.007455:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434725.029997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434725.158049:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434725.158069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434725.158073:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434725.158084:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7094 00000100:00080000:0.0:1789434725.158094:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434725.158106:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434725.158153:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913855488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434725.158183:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434725.158187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434725.158189:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434725.158193:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7094 00000100:00080000:0.0:1789434725.158197:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434725.158202:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434725.158212:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434725.158224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913855616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434725.158233:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434725.158236:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434725.158238:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434725.158241:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7094 00000100:00080000:0.0:1789434725.158244:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434725.158248:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434725.158252:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434725.158262:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913855744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434725.158272:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434725.158275:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434725.158278:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434725.158281:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7094 00000100:00080000:0.0:1789434725.158285:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789434725.158289:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913855488:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434725.158292:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434725.158301:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434725.158312:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913855872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434725.158377:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434725.158407:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855744:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434725.158433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855872:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434726.054067:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434727.080490:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434728.103897:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434728.231816:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434728.231891:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434728.231906:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434728.231938:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434728.231947:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434728.231954:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434728.232061:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434725/real 1789434728] req@ffff9811c8cd6a00 x1876350913855872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434728.232091:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913855872/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434728.232103:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434728.232107:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434728.232111:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855872:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789434728.232118:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434725/real 1789434728] req@ffff9811c8cd7480 x1876350913855488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434728.232130:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913855488/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434728.232137:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434728.232139:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434728.232142:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913855488:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434728.232155:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434725/real 1789434728] req@ffff9811c8cd7800 x1876350913855616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434728.232166:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913855616/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434728.232173:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434728.232176:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434728.232179:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434728.232185:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434725/real 1789434728] req@ffff9811c8cd5180 x1876350913855744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434728.232196:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913855744/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434728.232203:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434728.232205:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434728.232208:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913855744:192.168.203.128@tcp:38:kworker.0 00010000:00010000:3.0:1789434729.129413:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434730.150037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434730.278059:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434730.278068:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434730.278072:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434730.278082:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7099 00000100:00080000:2.0:1789434730.278090:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434730.278101:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434730.278134:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913856000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434730.278166:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434730.278170:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434730.278172:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434730.278176:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7099 00000100:00080000:2.0:1789434730.278179:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434730.278184:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434730.278193:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434730.278204:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913856128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434730.278213:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434730.278216:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434730.278218:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434730.278221:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7099 00000100:00080000:2.0:1789434730.278225:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434730.278229:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434730.278233:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434730.278242:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913856256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434730.278252:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434730.278255:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434730.278256:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434730.278260:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7099 00000100:00080000:2.0:1789434730.278264:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434730.278268:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434730.278272:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434730.278282:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913856384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434730.278363:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913856000:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434730.278418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434730.278452:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434730/real 1789434730] req@ffff9811d2100e00 x1876350913856000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434730.278469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913856000/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434730.278480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434730.278484:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434730.278488:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913856000:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434730.278495:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434730.278516:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434730.278528:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434730/real 1789434730] req@ffff9811d2101f80 x1876350913856128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434730.278539:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913856128/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434730.278553:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434730.278556:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434730.278559:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434730.278564:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856256:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434730.278581:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434730.278592:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434730/real 1789434730] req@ffff9811d2101180 x1876350913856256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434730.278603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913856256/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434730.278611:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434730.278614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434730.278617:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856256:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434730.278622:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856384:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434730.278640:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434730.278651:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434730/real 1789434730] req@ffff9811d2101880 x1876350913856384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434730.278662:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913856384/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434730.278670:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434730.278673:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434730.278677:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856384:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434731.174268:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434732.199197:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434733.222407:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434734.246432:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434735.270399:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434735.398046:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434735.398060:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434735.398065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434735.398078:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7104 00000100:00080000:0.0:1789434735.398090:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434735.398104:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434735.398160:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913856512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434735.398249:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913856512:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434735.398326:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789434735.398360:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434735.398365:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434735.398367:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434735.398373:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7104 00000100:00080000:0.0:1789434735.398377:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434735.398382:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434735.398394:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434735.398407:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913856640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434735.398518:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434735/real 1789434735] req@ffff9811c8cd5180 x1876350913856512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434735.398530:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789434735.398540:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913856512/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434735.398559:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434735.398569:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434735.398572:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434735.398579:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00080000:0.0:1789434735.398584:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7104 00000100:00100000:1.0:1789434735.398586:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913856512:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434735.398596:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789434735.398620:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856640:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434735.398621:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434735.398640:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789434735.398660:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434735.398676:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434735/real 1789434735] req@ffff9811c8cd7800 x1876350913856640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434735.398690:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913856768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434735.398696:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913856640/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434735.398730:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434735.398737:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00080000:0.0:1789434735.398739:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789434735.398744:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856640:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434735.398747:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434735.398755:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434735.398762:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7104 00000100:00080000:0.0:1789434735.398767:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789434735.398770:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856768:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789434735.398779:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434735.398786:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:1.0:1789434735.398801:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434735.398813:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913856896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434735.398820:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434735/real 1789434735] req@ffff9811c8cd7480 x1876350913856768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434735.398900:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913856768/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434735.398909:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434735.398913:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434735.398917:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856768:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434735.398936:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856896:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434735.398963:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434735.398976:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434735/real 1789434735] req@ffff9811c8cd6a00 x1876350913856896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434735.398999:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913856896/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434735.399008:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434735.399012:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434735.399015:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913856896:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434736.294077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434737.318025:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434738.342016:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434739.367174:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434740.390386:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434740.517994:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434740.518004:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434740.518007:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434740.518016:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7110 00000100:00080000:2.0:1789434740.518024:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434740.518034:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434740.518067:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913857024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434740.518103:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434740.518107:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434740.518109:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434740.518113:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7110 00000100:00080000:2.0:1789434740.518117:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434740.518122:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434740.518132:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434740.518145:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913857152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434740.518155:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434740.518158:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434740.518160:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434740.518163:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7110 00000100:00080000:2.0:1789434740.518166:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434740.518170:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434740.518174:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434740.518183:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913857280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434740.518193:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434740.518196:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434740.518198:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434740.518201:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7110 00000100:00080000:2.0:1789434740.518204:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434740.518207:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434740.518212:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434740.518223:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913857408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434740.518302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913857024:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434740.518357:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434740.518395:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434740/real 1789434740] req@ffff9811d2101500 x1876350913857024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434740.518417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913857024/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434740.518426:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434740.518442:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434740.518445:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913857024:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434740.518452:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434740.518474:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434740.518486:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434740/real 1789434740] req@ffff9811d2102a00 x1876350913857152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434740.518497:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913857152/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434740.518505:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434740.518508:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434740.518510:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434740.518515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857280:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434740.518532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434740.518542:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434740/real 1789434740] req@ffff9811d2103b80 x1876350913857280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434740.518553:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913857280/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434740.518565:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434740.518568:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434740.518571:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857280:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434740.518576:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857408:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434740.518595:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434740.518604:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434740/real 1789434740] req@ffff9811d2101c00 x1876350913857408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434740.518615:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913857408/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434740.518622:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434740.518625:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434740.518641:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857408:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434741.414058:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434742.438043:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434743.465143:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434744.486039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434745.510072:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434745.638040:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434745.638051:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434745.638055:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434745.638065:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7115 00000100:00080000:1.0:1789434745.638080:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434745.638094:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434745.638134:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913857536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434745.638168:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434745.638173:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434745.638175:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434745.638179:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7115 00000100:00080000:1.0:1789434745.638183:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434745.638187:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434745.638198:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434745.638209:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913857664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434745.638219:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434745.638221:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434745.638223:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434745.638226:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7115 00000100:00080000:1.0:1789434745.638229:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434745.638232:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434745.638237:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434745.638246:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913857792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434745.638256:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434745.638259:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434745.638266:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434745.638269:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7115 00000100:00080000:1.0:1789434745.638272:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434745.638276:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434745.638281:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434745.638291:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913857920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434745.638505:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913857536:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434745.638550:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434745.638578:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434745/real 1789434745] req@ffff9811d0bdc700 x1876350913857536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434745.638589:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913857536/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434745.638596:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434745.638599:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434745.638601:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913857536:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434745.638607:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434745.638619:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434745.638626:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434745/real 1789434745] req@ffff9811d0bdc000 x1876350913857664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434745.638633:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913857664/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434745.638637:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434745.638639:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434745.638641:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434745.638644:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857792:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434745.638654:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434745.638659:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434745/real 1789434745] req@ffff9811d0bdd880 x1876350913857792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434745.638665:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913857792/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434745.638669:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434745.638671:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434745.638672:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857792:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434745.638675:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857920:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434745.638684:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434745.638688:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434745/real 1789434745] req@ffff9811d0bde680 x1876350913857920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434745.638695:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913857920/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434745.638700:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434745.638701:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434745.638703:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913857920:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434746.534279:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434747.558543:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434748.582086:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434749.606056:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434750.630196:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434750.758189:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434750.758199:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434750.758203:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434750.758213:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7120 00000100:00080000:2.0:1789434750.758224:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434750.758235:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434750.758368:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913858048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434750.758394:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434750.758397:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434750.758399:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434750.758403:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7120 00000100:00080000:2.0:1789434750.758407:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434750.758412:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434750.758420:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434750.758432:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913858176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434750.758439:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434750.758442:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434750.758444:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434750.758447:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7120 00000100:00080000:2.0:1789434750.758450:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434750.758455:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434750.758460:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434750.758470:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913858304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434750.758479:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434750.758483:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434750.758484:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434750.758488:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7120 00000100:00080000:2.0:1789434750.758491:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434750.758494:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434750.758499:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434750.758511:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913858432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434750.759424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913858048:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434750.759516:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434750.759570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434750/real 1789434750] req@ffff9811d2102d80 x1876350913858048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434750.759591:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913858048/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434750.759604:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434750.759608:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434750.759613:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913858048:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434750.759620:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434750.759644:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434750.759656:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434750/real 1789434750] req@ffff9811d2103800 x1876350913858176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434750.759667:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913858176/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434750.759680:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434750.759684:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434750.759687:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434750.759693:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858304:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434750.759711:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434750.759722:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434750/real 1789434750] req@ffff9811d2100000 x1876350913858304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434750.759733:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913858304/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434750.759741:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434750.759744:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434750.759747:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858304:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434750.759753:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858432:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434750.759774:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434750.759784:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434750/real 1789434750] req@ffff9811d2101c00 x1876350913858432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434750.759794:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913858432/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434750.759802:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434750.759806:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434750.759809:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858432:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434751.654061:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434752.678941:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434753.702040:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434754.726015:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434755.750495:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434755.878026:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434755.878037:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434755.878041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434755.878051:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7125 00000100:00080000:0.0:1789434755.878059:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434755.878187:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434755.878227:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913858560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434755.878264:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434755.878268:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434755.878270:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434755.878274:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7125 00000100:00080000:0.0:1789434755.878278:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434755.878283:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434755.878292:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434755.878304:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913858688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434755.878312:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434755.878315:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434755.878316:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434755.878319:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7125 00000100:00080000:0.0:1789434755.878322:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434755.878327:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434755.878330:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434755.878339:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913858816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434755.878347:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434755.878350:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434755.878351:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434755.878355:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7125 00000100:00080000:0.0:1789434755.878358:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434755.878362:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434755.878367:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434755.878376:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913858944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434755.878490:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913858560:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434755.878550:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434755.878587:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434755/real 1789434755] req@ffff9811c8cd4000 x1876350913858560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434755.878603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913858560/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434755.878612:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434755.878616:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434755.878619:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913858560:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434755.878625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434755.878645:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434755.878656:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434755/real 1789434755] req@ffff9811c8cd6300 x1876350913858688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434755.878666:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913858688/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434755.878674:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434755.878677:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434755.878679:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434755.878684:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858816:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434755.878702:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434755.878712:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434755/real 1789434755] req@ffff9811c8cd7b80 x1876350913858816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434755.878722:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913858816/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434755.878729:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434755.878731:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434755.878734:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858816:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434755.878739:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858944:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434755.878758:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434755.878768:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434755/real 1789434755] req@ffff9811c8cd4a80 x1876350913858944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434755.878778:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913858944/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434755.878785:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434755.878787:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434755.878790:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913858944:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434756.774055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434757.798144:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434758.822237:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434759.846037:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434760.870049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434760.997942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434760.997951:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434760.997955:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434760.997964:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7130 00000100:00080000:0.0:1789434760.997974:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434760.997985:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434760.998020:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913859072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434760.998054:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434760.998058:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434760.998060:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434760.998064:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7130 00000100:00080000:0.0:1789434760.998068:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434760.998073:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434760.998082:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434760.998095:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913859200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434760.998104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434760.998107:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434760.998109:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434760.998112:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7130 00000100:00080000:0.0:1789434760.998115:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434760.998119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434760.998123:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434760.998130:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913859328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434760.998139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434760.998143:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434760.998144:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434760.998148:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7130 00000100:00080000:0.0:1789434760.998150:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434760.998155:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434760.998159:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434760.998170:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913859456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434760.998720:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913859072:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434760.998774:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434760.998820:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434760/real 1789434760] req@ffff9811c8cd5f80 x1876350913859072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434760.998864:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913859072/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434760.998874:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434760.998878:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434760.998881:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913859072:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434760.998887:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434760.998908:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434760.998919:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434760/real 1789434760] req@ffff9811c8cd6d80 x1876350913859200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434760.998930:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913859200/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434760.998937:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434760.998940:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434760.998943:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434760.998948:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859328:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434760.998962:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434760.998971:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434760/real 1789434760] req@ffff9811c8cd5500 x1876350913859328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434760.998981:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913859328/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434760.998993:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434760.998995:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434760.998998:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859328:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434760.999003:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859456:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434760.999017:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434760.999026:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434760/real 1789434760] req@ffff9811c8cd4a80 x1876350913859456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434760.999036:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913859456/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434760.999042:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434760.999045:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434760.999047:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859456:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434761.894008:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434762.918603:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434763.942052:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434764.966060:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434765.991453:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434766.118604:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434766.118614:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434766.118618:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434766.118628:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7135 00000100:00080000:0.0:1789434766.118639:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434766.118652:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434766.118688:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913859584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434766.118723:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434766.118727:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434766.118728:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434766.118733:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7135 00000100:00080000:0.0:1789434766.118737:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434766.118741:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434766.118751:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434766.118776:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913859712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434766.118785:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434766.118789:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434766.118790:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434766.118794:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7135 00000100:00080000:0.0:1789434766.118797:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434766.118802:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434766.118807:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434766.118817:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913859840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434766.118827:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434766.118862:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434766.118865:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434766.118869:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7135 00000100:00080000:0.0:1789434766.118873:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434766.118878:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434766.118883:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434766.118896:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913859968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434766.122822:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913859584:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434766.122927:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434766.122968:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434766/real 1789434766] req@ffff9811c8cd7b80 x1876350913859584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434766.122986:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913859584/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434766.122998:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434766.123003:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434766.123008:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913859584:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434766.123015:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859712:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434766.123039:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434766.123051:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434766/real 1789434766] req@ffff9811c8cd6300 x1876350913859712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434766.123063:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913859712/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434766.123071:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434766.123074:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434766.123077:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434766.123083:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859840:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434766.123103:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434766.123114:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434766/real 1789434766] req@ffff9811c8cd4000 x1876350913859840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434766.123125:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913859840/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434766.123133:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434766.123136:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434766.123139:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859840:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434766.123146:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859968:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434766.123182:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434766.123193:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434766/real 1789434766] req@ffff9811c8cd6a00 x1876350913859968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434766.123205:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913859968/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434766.123212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434766.123216:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434766.123219:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913859968:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434767.014066:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434768.038054:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434769.062049:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434770.086782:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434771.110140:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434771.238028:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434771.238038:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434771.238042:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434771.238052:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7140 00000100:00080000:2.0:1789434771.238062:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434771.238071:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434771.238100:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913860096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434771.238123:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434771.238133:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434771.238134:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434771.238139:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7140 00000100:00080000:2.0:1789434771.238143:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434771.238147:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434771.238157:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434771.238178:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913860224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434771.238192:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434771.238196:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434771.238198:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434771.238202:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7140 00000100:00080000:2.0:1789434771.238205:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434771.238209:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434771.238214:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434771.238224:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913860352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434771.238234:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434771.238237:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434771.238239:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434771.238243:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7140 00000100:00080000:2.0:1789434771.238246:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434771.238249:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434771.238254:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434771.238264:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913860480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434771.238317:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913860096:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434771.238392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434771.238434:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434771/real 1789434771] req@ffff9811d2103b80 x1876350913860096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434771.238450:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913860096/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434771.238461:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434771.238465:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434771.238468:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913860096:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434771.238473:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434771.238490:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434771.238506:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434771/real 1789434771] req@ffff9811d2102a00 x1876350913860224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434771.238517:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913860224/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434771.238530:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434771.238533:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434771.238536:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434771.238542:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860352:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434771.238561:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434771.238571:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434771/real 1789434771] req@ffff9811d2101500 x1876350913860352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434771.238582:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913860352/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434771.238589:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434771.238592:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434771.238595:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860352:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434771.238601:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860480:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434771.238621:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434771.238631:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434771/real 1789434771] req@ffff9811d2101880 x1876350913860480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434771.238642:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913860480/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434771.238649:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434771.238651:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434771.238654:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860480:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434772.134162:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434773.158432:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434774.181997:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434775.206004:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434776.231783:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434776.358554:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434776.358569:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434776.358574:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434776.358604:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7145 00000100:00080000:3.0:1789434776.358616:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434776.358698:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434776.358769:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913860608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434776.358806:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434776.358811:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434776.358813:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434776.358818:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7145 00000100:00080000:3.0:1789434776.358822:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434776.358827:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434776.358863:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434776.358878:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913860736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434776.358893:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913860608:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434776.358915:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434776.358919:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434776.358921:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434776.358926:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7145 00000100:00080000:3.0:1789434776.358930:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434776.358936:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434776.358941:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434776.358954:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913860864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434776.358964:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434776.358968:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434776.358970:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434776.358974:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7145 00000100:00080000:3.0:1789434776.358977:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434776.358984:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789434776.358987:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789434776.358991:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434776.359004:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913860992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434776.359038:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434776/real 1789434776] req@ffff9811d1bc5880 x1876350913860608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434776.359057:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913860608/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434776.359068:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434776.359071:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434776.359075:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913860608:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434776.359099:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434776.359120:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434776.359132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434776/real 1789434776] req@ffff9811d1bc7800 x1876350913860736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434776.359145:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913860736/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434776.359152:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434776.359155:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434776.359158:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434776.359164:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860864:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434776.359183:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434776.359195:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434776/real 1789434776] req@ffff9811d1bc5180 x1876350913860864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434776.359206:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913860864/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434776.359213:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434776.359216:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434776.359219:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860864:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434776.359225:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860992:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434776.359243:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434776.359253:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434776/real 1789434776] req@ffff9811d1bc6a00 x1876350913860992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434776.359263:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913860992/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434776.359270:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434776.359273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434776.359277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913860992:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434777.254161:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434778.279347:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434779.302718:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434780.326015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434781.350121:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434781.477927:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434781.477936:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434781.477941:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434781.477950:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7151 00000100:00080000:3.0:1789434781.477960:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434781.477972:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434781.478010:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913861120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434781.478035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434781.478038:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434781.478040:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434781.478044:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7151 00000100:00080000:3.0:1789434781.478048:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434781.478053:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434781.478062:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434781.478076:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913861248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434781.478086:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434781.478089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434781.478091:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434781.478094:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7151 00000100:00080000:3.0:1789434781.478097:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434781.478102:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434781.478106:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434781.478117:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913861376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434781.478132:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434781.478135:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434781.478137:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434781.478140:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7151 00000100:00080000:3.0:1789434781.478144:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434781.478148:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434781.478153:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434781.478372:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913861504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434781.478399:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913861120:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434781.478457:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434781.478491:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434781/real 1789434781] req@ffff9811d1bc7480 x1876350913861120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434781.478508:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913861120/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434781.478518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434781.478522:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434781.478526:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913861120:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434781.478532:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434781.478555:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434781.478567:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434781/real 1789434781] req@ffff9811d1bc6a00 x1876350913861248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434781.478579:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913861248/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434781.478587:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434781.478590:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434781.478593:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434781.478599:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861376:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434781.478618:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434781.478630:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434781/real 1789434781] req@ffff9811d1bc5180 x1876350913861376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434781.478652:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913861376/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434781.478661:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434781.478664:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434781.478667:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861376:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434781.478674:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861504:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434781.478694:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434781.478704:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434781/real 1789434781] req@ffff9811d1bc7800 x1876350913861504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434781.478714:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913861504/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434781.478722:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434781.478725:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434781.478728:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861504:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434782.374029:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434783.398292:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434784.422182:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434785.446057:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434786.470368:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434786.597937:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434786.597947:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434786.597951:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434786.597962:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7156 00000100:00080000:2.0:1789434786.597971:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434786.597983:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434786.598018:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913861632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434786.598094:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913861632:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434786.598162:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434786.598190:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434786.598192:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434786.598194:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434786.598197:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7156 00000100:00080000:2.0:1789434786.598200:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434786.598204:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434786.598212:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434786.598225:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913861760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434786.598280:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434786/real 1789434786] req@ffff9811d2101180 x1876350913861632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434786.598296:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913861632/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434786.598305:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434786.598309:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434786.598313:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913861632:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434786.598333:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434786.598356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434786.598369:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434786/real 1789434786] req@ffff9811d2101f80 x1876350913861760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434786.598382:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913861760/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434786.598389:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434786.598393:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434786.598396:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861760:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434786.598465:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434786.598468:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434786.598471:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434786.598475:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7156 00000100:00080000:2.0:1789434786.598480:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434786.598486:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434786.598491:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434786.598504:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913861888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434786.598515:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434786.598518:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434786.598519:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434786.598523:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7156 00000100:00080000:2.0:1789434786.598526:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434786.598530:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434786.598534:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434786.598544:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913862016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434786.598675:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861888:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434786.598693:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434786.598702:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434786/real 1789434786] req@ffff9811d2100e00 x1876350913861888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434786.598709:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913861888/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434786.598714:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434786.598716:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434786.598718:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913861888:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434786.598724:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862016:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434786.598741:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434786.598751:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434786/real 1789434786] req@ffff9811d2101f80 x1876350913862016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434786.598762:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913862016/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434786.598776:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434786.598779:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434786.598782:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862016:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434787.494533:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434788.519243:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434789.542433:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434790.566035:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434791.590224:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434791.718012:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434791.718022:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434791.718026:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434791.718036:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7161 00000100:00080000:2.0:1789434791.718044:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434791.718056:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434791.718091:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913862144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434791.718174:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913862144:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434791.718277:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434791.718282:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434791.718284:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434791.718289:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7161 00000100:00080000:2.0:1789434791.718294:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434791.718300:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434791.718310:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434791.718323:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913862272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434791.720476:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862272:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434791.720826:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434791.720841:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434791.720845:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434791.720850:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7161 00000100:00080000:2.0:1789434791.720856:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434791.720865:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434791.720874:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434791.720897:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913862400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434791.720949:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862400:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434791.720996:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434791.721000:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434791.721002:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434791.721005:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7161 00000100:00080000:2.0:1789434791.721009:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434791.721014:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434791.721019:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434791.721033:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913862528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434791.721066:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862528:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434792.614045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434793.638003:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434794.662213:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789434794.793628:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789434794.793739:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789434794.793756:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434794.798653:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434791/real 1789434794] req@ffff9811d2101180 x1876350913862144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434816 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434794.798687:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913862144/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434816 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434794.798700:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434794.798705:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434794.798709:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913862144:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434794.798896:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434794.798926:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434794.798935:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434794.799053:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434791/real 1789434794] req@ffff9811d2102a00 x1876350913862528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434794.799071:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913862528/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434794.799081:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434794.799085:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434794.799088:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862528:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789434794.799107:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434791/real 1789434794] req@ffff9811d2101500 x1876350913862400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434794.799120:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913862400/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434794.799128:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434794.799131:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434794.799135:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862400:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434794.799141:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434791/real 1789434794] req@ffff9811d2101880 x1876350913862272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434794.799163:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913862272/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434794.799170:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434794.799174:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434794.799177:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862272:192.168.203.128@tcp:38:kworker.0 00010000:00010000:2.0:1789434795.690137:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434796.710194:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434796.838002:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434796.838009:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434796.838013:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434796.838022:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7166 00000100:00080000:1.0:1789434796.838033:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434796.838044:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434796.838079:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913862656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434796.838165:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913862656:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434796.838220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434796.838250:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434796.838254:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434796.838256:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434796.838260:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7166 00000100:00080000:1.0:1789434796.838264:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434796.838269:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434796.838278:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434796.838290:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913862784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434796.838470:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434796/real 1789434796] req@ffff9811d0bdf480 x1876350913862656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434796.838493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913862656/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434796.838502:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434796.838505:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434796.838509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913862656:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434796.838538:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434796.838563:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434796.838575:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434796/real 1789434796] req@ffff9811d0bdfb80 x1876350913862784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434796.838586:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913862784/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434796.838594:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434796.838596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434796.838599:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862784:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789434796.840069:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434796.840076:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434796.840079:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434796.840088:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7166 00000100:00080000:1.0:1789434796.840096:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434796.840106:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434796.840117:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434796.841140:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913862912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434796.841160:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434796.841163:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434796.841166:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434796.841171:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7166 00000100:00080000:1.0:1789434796.841177:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434796.841185:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434796.841303:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434796.841320:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913863040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434796.842006:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862912:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434796.842067:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434796.842096:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434796/real 1789434796] req@ffff9811d0bded80 x1876350913862912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434796.842112:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913862912/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434796.842224:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434796.842230:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434796.842234:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913862912:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434796.842241:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863040:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434796.842262:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434796.842274:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434796/real 1789434796] req@ffff9811d0bdc380 x1876350913863040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434796.842322:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913863040/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434796.842339:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434796.842342:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434796.842345:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863040:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434797.734014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434798.758409:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434799.782052:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434800.806017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434801.830141:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434801.957955:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434801.957979:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434801.957983:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434801.957992:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7171 00000100:00080000:3.0:1789434801.958001:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434801.958012:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434801.958047:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913863168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434801.958072:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434801.958075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434801.958077:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434801.958081:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7171 00000100:00080000:3.0:1789434801.958084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434801.958089:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434801.958098:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434801.958111:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913863168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434801.958113:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913863296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434801.958128:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434801.958131:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434801.958133:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434801.958136:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7171 00000100:00080000:3.0:1789434801.958140:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434801.958144:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434801.958149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434801.958160:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913863424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434801.958180:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789434801.958186:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434801.958187:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434801.958191:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434801.958195:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7171 00000100:00080000:3.0:1789434801.958199:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434801.958205:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434801.958210:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434801.958225:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913863552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434801.958229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434801/real 1789434801] req@ffff9811d1bc5880 x1876350913863168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434801.958256:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913863168/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434801.958267:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434801.958271:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434801.958275:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913863168:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434801.958296:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434801.958323:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434801.958336:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434801/real 1789434801] req@ffff9811d1bc6680 x1876350913863296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434801.958348:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913863296/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434801.958382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434801.958386:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434801.958389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434801.958395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863424:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434801.958415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434801.958425:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434801/real 1789434801] req@ffff9811d1bc5c00 x1876350913863424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434801.958437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913863424/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434801.958445:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434801.958448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434801.958452:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863424:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434801.958458:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863552:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434801.958478:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434801.958489:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434801/real 1789434801] req@ffff9811d1bc4000 x1876350913863552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434801.958512:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913863552/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434801.958521:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434801.958524:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434801.958527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863552:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434802.854335:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434803.878297:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434804.902069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434805.926152:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434806.950084:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434807.078508:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434807.078518:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434807.078522:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434807.078534:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7176 00000100:00080000:0.0:1789434807.078543:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434807.078553:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434807.078595:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913863680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434807.078628:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434807.078631:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434807.078633:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434807.078637:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7176 00000100:00080000:0.0:1789434807.078639:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434807.078643:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434807.078653:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434807.078663:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913863808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434807.078677:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434807.078680:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434807.078682:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434807.078685:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7176 00000100:00080000:0.0:1789434807.078688:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434807.078691:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434807.078694:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434807.078701:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913863936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434807.078708:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434807.078711:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434807.078713:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434807.078715:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7176 00000100:00080000:0.0:1789434807.078718:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434807.078722:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434807.078725:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434807.078733:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913864064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434807.078817:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913863680:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434807.078900:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434807.078940:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434807/real 1789434807] req@ffff9811c8cd7480 x1876350913863680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434807.078956:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913863680/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434807.078965:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434807.078969:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434807.078972:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913863680:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434807.078978:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434807.078997:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434807.079008:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434807/real 1789434807] req@ffff9811c8cd7800 x1876350913863808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434807.079023:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913863808/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434807.079029:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434807.079032:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434807.079035:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434807.079040:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863936:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434807.079059:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434807.079069:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434807/real 1789434807] req@ffff9811c8cd5180 x1876350913863936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434807.079081:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913863936/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434807.079088:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434807.079091:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434807.079094:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913863936:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434807.079100:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864064:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434807.079121:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434807.079132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434807/real 1789434807] req@ffff9811c8cd7100 x1876350913864064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434807.079142:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913864064/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434807.079150:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434807.079152:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434807.079155:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864064:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434807.973994:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434809.000620:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434810.022133:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434811.048116:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434812.073322:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434812.198002:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434812.198011:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434812.198015:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434812.198026:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7181 00000100:00080000:1.0:1789434812.198036:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434812.198047:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434812.198083:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913864192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434812.198119:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434812.198123:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434812.198125:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434812.198130:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7181 00000100:00080000:1.0:1789434812.198134:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434812.198139:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434812.198148:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434812.198161:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913864192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434812.198162:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913864320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434812.198177:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434812.198180:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434812.198182:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434812.198185:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7181 00000100:00080000:1.0:1789434812.198188:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434812.198192:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434812.198196:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434812.198207:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913864448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434812.198219:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434812.198222:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434812.198224:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434812.198228:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7181 00000100:00080000:1.0:1789434812.198232:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434812.198236:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434812.198241:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789434812.198244:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434812.198258:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913864576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434812.198284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434812/real 1789434812] req@ffff9811d0bdd500 x1876350913864192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434812.198320:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913864192/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434812.198331:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434812.198335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434812.198339:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913864192:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434812.198360:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434812.198392:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434812.198405:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434812/real 1789434812] req@ffff9811d0bddc00 x1876350913864320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434812.198429:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913864320/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434812.198437:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434812.198450:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434812.198454:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434812.198461:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864448:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434812.198483:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434812.198498:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434812/real 1789434812] req@ffff9811d0bdd180 x1876350913864448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434812.198509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913864448/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434812.198532:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434812.198536:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434812.198539:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864448:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434812.198545:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864576:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434812.198563:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434812.198573:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434812/real 1789434812] req@ffff9811d0bdc380 x1876350913864576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434812.198584:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913864576/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434812.198592:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434812.198595:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434812.198598:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864576:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434813.094067:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434814.118032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434815.142412:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434816.166361:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434817.190175:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434817.317958:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434817.317968:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434817.317972:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434817.317982:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7186 00000100:00080000:2.0:1789434817.317991:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434817.318003:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434817.318039:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913864704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434817.318069:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434817.318073:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434817.318075:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434817.318079:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7186 00000100:00080000:2.0:1789434817.318083:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434817.318090:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434817.318100:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434817.318108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913864704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434817.318128:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913864832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434817.318144:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434817.318148:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434817.318150:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434817.318153:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7186 00000100:00080000:2.0:1789434817.318157:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434817.318162:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434817.318167:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434817.318178:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913864960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434817.318187:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434817.318201:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434817.318205:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434817.318208:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434817.318213:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7186 00000100:00080000:2.0:1789434817.318217:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434817.318222:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434817.318229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434817/real 1789434817] req@ffff9811d2103b80 x1876350913864704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789434817.318229:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434817.318245:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913865088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434817.318251:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913864704/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434817.318268:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434817.318273:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434817.318277:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913864704:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434817.318297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434817.318316:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434817.318326:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434817/real 1789434817] req@ffff9811d2101c00 x1876350913864832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434817.318337:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913864832/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434817.318355:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434817.318358:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434817.318361:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434817.318366:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864960:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434817.318393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434817.318403:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434817/real 1789434817] req@ffff9811d2100000 x1876350913864960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434817.318413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913864960/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434817.318421:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434817.318424:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434817.318427:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913864960:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434817.318432:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865088:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434817.318446:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434817.318453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434817/real 1789434817] req@ffff9811d2103800 x1876350913865088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434817.318464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913865088/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434817.318471:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434817.318474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434817.318476:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865088:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434818.214054:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434819.238049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434820.262224:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434821.286038:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434822.309998:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434822.437951:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434822.437962:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434822.437965:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434822.437975:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7192 00000100:00080000:2.0:1789434822.437986:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434822.437998:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434822.438034:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913865216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434822.438068:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434822.438072:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434822.438074:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434822.438078:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7192 00000100:00080000:2.0:1789434822.438083:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434822.438087:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434822.438096:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434822.438110:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913865344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434822.438124:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434822.438127:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434822.438129:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434822.438132:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7192 00000100:00080000:2.0:1789434822.438136:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434822.438140:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434822.438145:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434822.438154:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913865472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434822.438163:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434822.438167:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434822.438168:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434822.438171:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7192 00000100:00080000:2.0:1789434822.438174:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434822.438178:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434822.438182:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434822.438191:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913865600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434822.438286:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913865216:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434822.438354:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434822.438390:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434822/real 1789434822] req@ffff9811d2102d80 x1876350913865216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434822.438406:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913865216/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434822.438416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434822.438419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434822.438423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913865216:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434822.438429:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434822.438462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434822.438474:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434822/real 1789434822] req@ffff9811d2103800 x1876350913865344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434822.438485:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913865344/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434822.438497:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434822.438500:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434822.438504:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434822.438509:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865472:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434822.438525:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434822.438534:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434822/real 1789434822] req@ffff9811d2100000 x1876350913865472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434822.438545:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913865472/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434822.438552:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434822.438555:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434822.438558:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865472:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434822.438564:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865600:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434822.438580:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434822.438590:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434822/real 1789434822] req@ffff9811d2101c00 x1876350913865600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434822.438600:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913865600/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434822.438607:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434822.438610:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434822.438614:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865600:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434823.334031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434824.358289:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434825.382014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434826.406989:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434827.430022:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434827.557959:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434827.557969:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434827.557974:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434827.558037:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7197 00000100:00080000:2.0:1789434827.558045:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434827.558054:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434827.558089:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913865728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434827.558166:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913865728:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434827.558179:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434827.558183:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434827.558184:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434827.558189:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7197 00000100:00080000:2.0:1789434827.558193:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434827.558197:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434827.558207:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434827.558219:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913865856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434827.558227:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434827.558227:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434827.558230:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434827.558231:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434827.558234:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7197 00000100:00080000:2.0:1789434827.558237:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434827.558240:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434827.558244:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434827.558252:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913865984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434827.558261:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434827/real 1789434827] req@ffff9811d2103b80 x1876350913865728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789434827.558268:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434827.558270:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434827.558272:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434827.558275:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7197 00000100:00100000:0.0:1789434827.558276:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913865728/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789434827.558278:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434827.558282:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00080000:0.0:1789434827.558285:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1789434827.558286:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:0.0:1789434827.558289:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434827.558293:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913865728:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434827.558294:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913866112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434827.558416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865856:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434827.558440:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434827.558453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434827/real 1789434827] req@ffff9811d2101880 x1876350913865856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434827.558465:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913865856/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434827.558472:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434827.558476:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434827.558479:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434827.558484:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865984:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434827.558503:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434827.558514:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434827/real 1789434827] req@ffff9811d2101500 x1876350913865984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434827.558525:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913865984/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434827.558531:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434827.558534:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434827.558537:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913865984:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434827.558542:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866112:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434827.558560:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434827.558570:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434827/real 1789434827] req@ffff9811d2102a00 x1876350913866112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434827.558580:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913866112/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434827.558587:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434827.558589:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434827.558592:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866112:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434828.454004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434829.478258:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434830.502360:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434831.526159:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434832.550192:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434832.679092:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434832.679102:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434832.679106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434832.679115:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7202 00000100:00080000:2.0:1789434832.679123:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434832.679134:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434832.679166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913866240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434832.679191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434832.679194:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434832.679196:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434832.679199:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7202 00000100:00080000:2.0:1789434832.679202:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434832.679206:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434832.679220:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434832.679231:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913866368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434832.679236:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913866240:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434832.679246:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434832.679250:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434832.679251:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434832.679255:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7202 00000100:00080000:2.0:1789434832.679258:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434832.679277:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434832.679283:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434832.679294:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913866496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434832.679305:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434832.679308:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434832.679310:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789434832.679313:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434832.679317:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7202 00000100:00080000:2.0:1789434832.679322:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434832.679327:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434832.679340:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434832.679360:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913866624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434832.679372:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434832/real 1789434832] req@ffff9811d2101180 x1876350913866240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434832.679393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913866240/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434832.679403:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434832.679407:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434832.679411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913866240:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434832.679433:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866368:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434832.679454:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434832.679469:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434832/real 1789434832] req@ffff9811d2101f80 x1876350913866368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434832.679482:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913866368/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434832.679490:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434832.679494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434832.679498:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434832.679503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866496:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434832.679518:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434832.679540:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434832/real 1789434832] req@ffff9811d2100e00 x1876350913866496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434832.679551:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913866496/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434832.679558:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434832.679560:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434832.679563:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866496:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434832.679568:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866624:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434832.679586:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434832.679596:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434832/real 1789434832] req@ffff9811d2102a00 x1876350913866624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434832.679607:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913866624/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434832.679613:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434832.679616:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434832.679619:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866624:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434833.573983:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434834.598305:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434835.623447:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434836.646382:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434837.670017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434837.797966:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434837.797975:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434837.797979:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434837.797989:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7207 00000100:00080000:0.0:1789434837.797999:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434837.798020:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434837.798055:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913866752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434837.798091:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434837.798100:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434837.798102:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434837.798105:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7207 00000100:00080000:0.0:1789434837.798109:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434837.798114:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434837.798124:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434837.798135:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913866880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434837.798145:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434837.798148:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434837.798149:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434837.798153:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7207 00000100:00080000:0.0:1789434837.798156:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434837.798160:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434837.798164:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434837.798173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913867008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434837.798182:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434837.798185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434837.798187:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434837.798190:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7207 00000100:00080000:0.0:1789434837.798193:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434837.798197:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434837.798200:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434837.798211:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913867136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434837.798270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913866752:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434837.798329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434837.798362:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434837/real 1789434837] req@ffff9811c8cd6680 x1876350913866752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434837.798378:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913866752/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434837.798388:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434837.798392:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434837.798395:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913866752:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434837.798401:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434837.798420:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434837.798431:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434837/real 1789434837] req@ffff9811c8cd4700 x1876350913866880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434837.798442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913866880/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434837.798450:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434837.798453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434837.798457:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913866880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434837.798462:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867008:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434837.798481:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434837.798491:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434837/real 1789434837] req@ffff9811c8cd5c00 x1876350913867008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434837.798502:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913867008/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434837.798510:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434837.798645:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434837.798651:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867008:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434837.798658:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867136:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434837.798681:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434837.798692:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434837/real 1789434837] req@ffff9811c8cd7100 x1876350913867136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434837.798702:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913867136/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434837.798714:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434837.798716:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434837.798719:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867136:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434838.693999:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434839.718006:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434840.743267:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434841.766047:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434842.790050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434842.918035:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434842.918043:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434842.918047:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434842.918056:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7212 00000100:00080000:3.0:1789434842.918063:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434842.918073:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434842.918106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913867264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434842.918132:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434842.918142:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434842.918143:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434842.918147:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7212 00000100:00080000:3.0:1789434842.918150:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434842.918154:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434842.918163:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434842.918173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913867392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434842.918181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434842.918184:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434842.918195:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434842.918199:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7212 00000100:00080000:3.0:1789434842.918201:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434842.918205:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434842.918209:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434842.918218:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913867520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434842.918227:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434842.918230:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434842.918231:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434842.918235:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7212 00000100:00080000:3.0:1789434842.918237:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434842.918241:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434842.918245:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434842.918254:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913867648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434842.918378:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913867264:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434842.918443:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434842.918478:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434842/real 1789434842] req@ffff9811d1bc5f80 x1876350913867264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434842.918493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913867264/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434842.918503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434842.918506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434842.918509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913867264:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434842.918515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434842.918534:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434842.918545:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434842/real 1789434842] req@ffff9811d1bc5500 x1876350913867392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434842.918557:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913867392/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434842.918564:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434842.918567:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434842.918571:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434842.918577:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867520:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434842.918593:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434842.918604:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434842/real 1789434842] req@ffff9811d1bc6300 x1876350913867520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434842.918615:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913867520/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434842.918622:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434842.918626:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434842.918629:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867520:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434842.918634:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867648:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434842.918652:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434842.918662:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434842/real 1789434842] req@ffff9811d1bc4700 x1876350913867648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434842.918674:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913867648/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434842.918682:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434842.918685:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434842.918688:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867648:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434843.814050:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434844.838763:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434845.862044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434846.888981:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434847.910867:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434848.037959:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434848.037969:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434848.037973:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434848.037983:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7217 00000100:00080000:2.0:1789434848.037992:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434848.038002:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434848.038035:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913867776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434848.038104:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913867776:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434848.038171:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434848.038199:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434848.038203:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434848.038205:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434848.038209:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7217 00000100:00080000:2.0:1789434848.038213:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434848.038218:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434848.038228:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434848.038241:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913867904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434848.038291:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434848/real 1789434848] req@ffff9811d2101500 x1876350913867776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434848.038308:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913867776/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434848.038318:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434848.038321:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434848.038325:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913867776:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434848.038344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434848.038366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434848.038377:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434848/real 1789434848] req@ffff9811d2101880 x1876350913867904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434848.038388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913867904/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434848.038396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434848.038399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434848.038402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913867904:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434848.038569:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434848.038573:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434848.038575:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434848.038579:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7217 00000100:00080000:2.0:1789434848.038583:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434848.038589:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434848.038594:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434848.038605:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913868032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434848.038646:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868032:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434848.038665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434848.038690:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434848.038694:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434848.038695:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434848.038698:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7217 00000100:00080000:2.0:1789434848.038702:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434848.038706:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434848.038711:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434848.038734:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913868160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434848.038798:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434848/real 1789434848] req@ffff9811d2103b80 x1876350913868032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434848.038808:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913868032/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434848.038820:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434848.038823:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434848.038825:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868032:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434848.038869:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868160:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434848.038889:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434848.038898:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434848/real 1789434848] req@ffff9811d2101c00 x1876350913868160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434848.038908:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913868160/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434848.038914:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434848.038917:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434848.038920:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868160:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434848.934019:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434849.958183:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434850.982054:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434852.006080:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434853.030474:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434853.157960:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434853.157968:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434853.157972:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434853.157982:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7222 00000100:00080000:2.0:1789434853.157992:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434853.158002:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434853.158035:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913868288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434853.158119:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913868288:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434853.158175:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434853.158206:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434853.158210:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434853.158212:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434853.158217:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7222 00000100:00080000:2.0:1789434853.158221:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434853.158227:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434853.158236:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434853.158249:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913868416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434853.161356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434853/real 1789434853] req@ffff9811d2100000 x1876350913868288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434853.161381:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913868288/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434853.161393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434853.161398:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434853.161402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913868288:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434853.161430:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434853.161477:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434853.161492:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434853/real 1789434853] req@ffff9811d2103800 x1876350913868416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434853.161504:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913868416/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434853.161512:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434853.162306:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434853.162312:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868416:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434853.162620:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434853.162626:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434853.162629:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434853.162634:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7222 00000100:00080000:2.0:1789434853.162641:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434853.162650:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434853.162672:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434853.162699:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913868544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434853.162713:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434853.162716:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434853.162717:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434853.162721:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7222 00000100:00080000:2.0:1789434853.162725:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434853.162729:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434853.162734:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434853.162745:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913868672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434853.162857:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868544:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434853.162894:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434853.162911:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434853/real 1789434853] req@ffff9811d2102d80 x1876350913868544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434853.162925:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913868544/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434853.162934:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434853.162937:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434853.162940:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868544:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434853.162946:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868672:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434853.162965:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434853.162975:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434853/real 1789434853] req@ffff9811d2103800 x1876350913868672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434853.163001:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913868672/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434853.163009:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434853.163012:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434853.163015:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868672:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434854.054151:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434855.078431:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434856.102035:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434857.126262:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434858.150122:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434858.278064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434858.278086:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434858.278091:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434858.278102:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7227 00000100:00080000:3.0:1789434858.278111:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434858.278122:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434858.278173:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913868800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434858.278201:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434858.278205:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434858.278207:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434858.278212:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7227 00000100:00080000:3.0:1789434858.278215:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434858.278220:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434858.278238:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434858.278246:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913868800:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434858.278256:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913868928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434858.278268:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434858.278272:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434858.278274:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434858.278278:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7227 00000100:00080000:3.0:1789434858.278282:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434858.278287:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434858.278292:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434858.278307:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913869056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434858.278316:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434858.278319:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434858.278321:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434858.278325:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7227 00000100:00080000:3.0:1789434858.278328:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434858.278333:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434858.278338:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434858.278349:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913869184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434858.278425:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434858.278450:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434858.278463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869184:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434859.174127:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434860.199298:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434861.222256:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789434861.350136:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:2.0:1789434861.350178:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789434861.350195:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789434861.350235:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789434861.350245:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:2.0:1789434861.350253:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434861.350583:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434858/real 1789434861] req@ffff9811d1bc4700 x1876350913868928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434861.350607:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913868928/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434861.350618:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434861.350623:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434861.350627:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913868928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434861.350635:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434858/real 1789434861] req@ffff9811d1bc6300 x1876350913869056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434861.350647:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913869056/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434861.350655:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434861.350658:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434861.350662:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869056:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434861.350668:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434858/real 1789434861] req@ffff9811d1bc5500 x1876350913869184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434861.350678:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913869184/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434861.350685:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434861.350688:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434861.350691:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869184:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789434861.350697:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434858/real 1789434861] req@ffff9811d1bc7100 x1876350913868800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434861.350708:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913868800/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434861.350715:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434861.350719:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434861.350722:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913868800:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789434862.245997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434863.270431:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434863.398408:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434863.398418:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434863.398422:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434863.398432:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7232 00000100:00080000:1.0:1789434863.398440:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434863.398452:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434863.398483:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913869312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434863.398515:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434863.398519:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434863.398521:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434863.398525:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7232 00000100:00080000:1.0:1789434863.398528:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434863.398533:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434863.398542:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434863.398555:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913869312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434863.398556:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913869440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434863.398572:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434863.398575:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434863.398577:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434863.398580:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7232 00000100:00080000:1.0:1789434863.398584:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434863.398588:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434863.398593:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434863.398604:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913869568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434863.398613:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789434863.398619:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434863.398623:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434863.398626:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434863.398630:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7232 00000100:00080000:1.0:1789434863.398634:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434863.398639:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434863.398645:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434863.398656:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913869696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434863.398723:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434863/real 1789434863] req@ffff9811d0bded80 x1876350913869312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434863.398743:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913869312/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434863.398753:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434863.398757:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434863.398760:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913869312:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434863.398783:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434863.398807:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434863.398861:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434863/real 1789434863] req@ffff9811d0bdfb80 x1876350913869440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434863.398881:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913869440/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434863.398890:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434863.398896:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434863.398899:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434863.398906:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869568:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434863.398925:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434863.398936:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434863/real 1789434863] req@ffff9811d0bdf480 x1876350913869568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434863.398951:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913869568/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434863.398958:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434863.398960:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434863.398963:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869568:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434863.398968:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869696:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434863.398987:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434863.398995:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434863/real 1789434863] req@ffff9811d0bde680 x1876350913869696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434863.399004:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913869696/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434863.399011:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434863.399013:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434863.399016:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869696:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434864.294173:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434865.318018:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434866.342118:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434867.366061:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434868.390561:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434868.518064:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434868.518074:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434868.518079:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434868.518089:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7238 00000100:00080000:1.0:1789434868.518097:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434868.518108:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434868.518140:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913869824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434868.518173:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434868.518177:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434868.518178:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434868.518182:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7238 00000100:00080000:1.0:1789434868.518186:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434868.518191:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434868.518201:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434868.518213:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913869952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434868.518222:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434868.518225:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434868.518227:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434868.518230:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7238 00000100:00080000:1.0:1789434868.518234:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434868.518238:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434868.518246:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434868.518257:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913870080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434868.518267:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434868.518270:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434868.518272:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434868.518275:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7238 00000100:00080000:1.0:1789434868.518277:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434868.518281:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434868.518285:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434868.518294:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913870208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434868.518435:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913869824:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434868.518491:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434868.518528:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434868/real 1789434868] req@ffff9811d0bdd880 x1876350913869824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434868.518545:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913869824/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434868.518560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434868.518564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434868.518568:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913869824:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434868.518574:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434868.518595:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434868.518619:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434868/real 1789434868] req@ffff9811d0bdc000 x1876350913869952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434868.518631:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913869952/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434868.518639:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434868.518642:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434868.518645:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913869952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434868.518651:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870080:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434868.518668:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434868.518677:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434868/real 1789434868] req@ffff9811d0bdc700 x1876350913870080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434868.518687:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913870080/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434868.518693:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434868.518695:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434868.518698:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870080:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434868.518704:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870208:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434868.518719:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434868.518727:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434868/real 1789434868] req@ffff9811d0bdea00 x1876350913870208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434868.518736:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913870208/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434868.518743:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434868.518746:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434868.518749:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870208:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434869.415157:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434870.438088:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434871.463087:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434872.486252:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434873.510021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434873.640063:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434873.640072:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434873.640076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434873.640087:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7243 00000100:00080000:2.0:1789434873.640096:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434873.640108:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434873.640147:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913870336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434873.640181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434873.640185:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434873.640186:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434873.640190:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7243 00000100:00080000:2.0:1789434873.640194:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434873.640198:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434873.640208:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434873.640232:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913870464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434873.640240:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434873.640243:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434873.640244:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434873.640247:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7243 00000100:00080000:2.0:1789434873.640250:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434873.640254:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434873.640258:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434873.640267:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913870592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434873.640277:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434873.640280:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434873.640282:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434873.640286:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7243 00000100:00080000:2.0:1789434873.640290:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434873.640295:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434873.640300:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434873.640303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913870336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434873.640312:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913870720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434873.640378:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434873.640428:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434873/real 1789434873] req@ffff9811d2100000 x1876350913870336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434873.640447:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913870336/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434873.640458:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434873.640462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434873.640465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913870336:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434873.640472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434873.640494:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434873.640506:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434873/real 1789434873] req@ffff9811d2101c00 x1876350913870464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434873.640519:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913870464/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434873.640527:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434873.640530:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434873.640533:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434873.640539:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870592:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434873.640555:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434873.640564:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434873/real 1789434873] req@ffff9811d2103b80 x1876350913870592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434873.640574:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913870592/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434873.640580:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434873.640583:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434873.640585:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870592:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434873.640606:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870720:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434873.640623:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434873.640633:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434873/real 1789434873] req@ffff9811d2101880 x1876350913870720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434873.640644:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913870720/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434873.640670:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434873.640673:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434873.640676:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870720:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434874.534181:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434875.558809:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434876.582333:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434877.607223:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434878.630035:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434878.758219:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434878.758229:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434878.758233:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434878.758242:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7248 00000100:00080000:1.0:1789434878.758251:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434878.758263:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434878.758298:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913870848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434878.758327:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434878.758331:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434878.758333:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434878.758337:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7248 00000100:00080000:1.0:1789434878.758341:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434878.758345:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434878.758355:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434878.758367:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913870976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434878.758376:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434878.758378:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434878.758380:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434878.758383:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7248 00000100:00080000:1.0:1789434878.758387:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434878.758391:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434878.758395:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434878.758405:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913871104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434878.758414:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434878.758417:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434878.758419:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434878.758423:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7248 00000100:00080000:1.0:1789434878.758426:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434878.758431:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434878.758435:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434878.758588:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913871232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434878.758662:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913870848:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434878.758720:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434878.759252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434878/real 1789434878] req@ffff9811d0bdf100 x1876350913870848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434878.759270:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913870848/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434878.759279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434878.759283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434878.759286:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913870848:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434878.759293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434878.759318:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434878.759329:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434878/real 1789434878] req@ffff9811d0bdf800 x1876350913870976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434878.759339:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913870976/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434878.759346:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434878.759348:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434878.759351:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913870976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434878.759357:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871104:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434878.759373:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434878.759383:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434878/real 1789434878] req@ffff9811d0bde300 x1876350913871104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434878.759393:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913871104/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434878.759400:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434878.759403:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434878.759406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871104:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434878.759411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871232:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434878.759428:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434878.759436:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434878/real 1789434878] req@ffff9811d0bdea00 x1876350913871232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434878.759445:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913871232/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434878.759453:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434878.759455:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434878.759458:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871232:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434879.654303:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434880.678047:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434881.702063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434882.726155:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434883.750110:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434883.877986:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434883.877994:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434883.877998:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434883.878005:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7253 00000100:00080000:3.0:1789434883.878013:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434883.878021:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434883.878049:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913871360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434883.878068:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434883.878070:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434883.878071:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434883.878074:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7253 00000100:00080000:3.0:1789434883.878076:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434883.878078:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434883.878086:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434883.878104:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913871488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434883.878107:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913871360:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434883.878119:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434883.878122:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434883.878124:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434883.878130:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7253 00000100:00080000:3.0:1789434883.878134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434883.878138:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434883.878144:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434883.878155:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913871616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434883.878163:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434883.878165:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434883.878166:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434883.878169:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7253 00000100:00080000:3.0:1789434883.878173:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434883.878176:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789434883.878184:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:3.0:1789434883.878186:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434883.878200:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913871744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434883.878227:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434883/real 1789434883] req@ffff9811d1bc5f80 x1876350913871360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434883.878255:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913871360/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434883.878271:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434883.878275:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434883.878279:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913871360:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434883.878300:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871488:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434883.878321:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434883.878339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434883/real 1789434883] req@ffff9811d1bc4000 x1876350913871488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434883.878355:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913871488/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434883.878362:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434883.878365:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434883.878368:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434883.878374:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871616:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434883.878393:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434883.878402:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434883/real 1789434883] req@ffff9811d1bc5c00 x1876350913871616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434883.878410:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913871616/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434883.878419:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434883.878421:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434883.878423:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871616:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434883.878427:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871744:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434883.878442:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434883.878451:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434883/real 1789434883] req@ffff9811d1bc6680 x1876350913871744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434883.878462:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913871744/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434883.878470:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434883.878473:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434883.878475:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913871744:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434884.777219:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434885.798170:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434886.822560:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434887.846110:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434888.871130:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434888.997980:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434888.997990:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434888.997995:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434888.998005:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7258 00000100:00080000:3.0:1789434888.998015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434888.998028:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434888.998066:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913871872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434888.998092:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434888.998097:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434888.998098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434888.998102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7258 00000100:00080000:3.0:1789434888.998106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434888.998111:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434888.998119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434888.998131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913872000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434888.998140:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434888.998143:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434888.998145:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434888.998149:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7258 00000100:00080000:3.0:1789434888.998152:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434888.998155:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434888.998159:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434888.998167:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913872128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434888.998177:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434888.998180:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434888.998182:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434888.998185:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7258 00000100:00080000:3.0:1789434888.998187:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434888.998190:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434888.998194:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434888.998204:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913872256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434888.999268:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913871872:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434888.999329:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434888.999370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434888/real 1789434888] req@ffff9811d1bc5880 x1876350913871872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434888.999388:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913871872/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434888.999398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434888.999402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434888.999405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913871872:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434888.999413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434888.999436:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434888.999448:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434888/real 1789434888] req@ffff9811d1bc7800 x1876350913872000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434888.999459:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913872000/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434888.999467:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434888.999471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434888.999474:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434888.999480:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872128:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434888.999496:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434888.999505:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434888/real 1789434888] req@ffff9811d1bc5180 x1876350913872128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434888.999516:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913872128/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434888.999523:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434888.999526:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434888.999529:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872128:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434888.999535:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872256:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434888.999554:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434888.999565:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434888/real 1789434888] req@ffff9811d1bc6a00 x1876350913872256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434888.999577:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913872256/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434888.999585:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434888.999587:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434888.999590:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872256:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434889.894320:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434890.918102:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434891.942166:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434892.966668:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434893.990028:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434894.120769:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434894.120780:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434894.120783:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434894.120793:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7263 00000100:00080000:2.0:1789434894.120803:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434894.120828:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434894.120905:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913872384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434894.120935:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434894.120938:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434894.120940:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434894.120945:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7263 00000100:00080000:2.0:1789434894.120949:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434894.120953:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434894.120962:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434894.120973:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913872512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434894.120982:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434894.120985:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434894.120987:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434894.120990:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7263 00000100:00080000:2.0:1789434894.120993:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434894.120997:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434894.121001:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434894.121010:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913872640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434894.121020:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434894.121023:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434894.121025:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434894.121028:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7263 00000100:00080000:2.0:1789434894.121031:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434894.121034:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434894.121038:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434894.121046:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913872768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434894.121160:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913872384:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434894.121214:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434894.121292:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434894/real 1789434894] req@ffff9811d2101500 x1876350913872384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434894.121311:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913872384/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434894.121321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434894.121324:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434894.121328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913872384:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434894.121335:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434894.121356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434894.121368:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434894/real 1789434894] req@ffff9811d2102a00 x1876350913872512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434894.121379:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913872512/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434894.121393:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434894.121396:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434894.121399:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434894.121404:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872640:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434894.121421:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434894.121430:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434894/real 1789434894] req@ffff9811d2100e00 x1876350913872640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434894.121441:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913872640/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434894.121449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434894.121452:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434894.121455:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872640:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434894.121460:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872768:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434894.121476:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434894.121485:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434894/real 1789434894] req@ffff9811d2101f80 x1876350913872768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434894.121493:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913872768/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434894.121500:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434894.121503:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434894.121506:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913872768:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434895.014066:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434896.038012:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434897.062031:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434898.086055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434899.110252:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434899.237970:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434899.237982:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434899.237987:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434899.237998:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7268 00000100:00080000:2.0:1789434899.238008:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434899.238022:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434899.238060:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913872896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434899.238088:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434899.238093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434899.238096:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434899.238102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7268 00000100:00080000:2.0:1789434899.238107:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434899.238113:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434899.238124:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434899.238128:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913872896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434899.238143:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913873024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434899.238155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434899.238158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434899.238160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434899.238165:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7268 00000100:00080000:2.0:1789434899.238169:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434899.238175:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434899.238180:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434899.238193:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913873152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434899.238205:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434899.238209:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434899.238211:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434899.238215:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7268 00000800:00000100:0.0:1789434899.238220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434899.238220:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434899.238231:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434899.238237:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434899.238254:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913873280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434899.238286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434899/real 1789434899] req@ffff9811d2101180 x1876350913872896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434899.238323:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913872896/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434899.238335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434899.238340:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434899.238344:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913872896:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434899.238380:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873024:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434899.238405:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434899.238417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434899/real 1789434899] req@ffff9811d2101f80 x1876350913873024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434899.238427:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913873024/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434899.238435:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434899.238438:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434899.238441:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434899.238447:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873152:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434899.238466:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434899.238477:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434899/real 1789434899] req@ffff9811d2100e00 x1876350913873152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434899.238487:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913873152/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434899.238494:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434899.238498:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434899.238500:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873152:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434899.238520:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873280:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434899.238541:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434899.238552:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434899/real 1789434899] req@ffff9811d2102a00 x1876350913873280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434899.238567:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913873280/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434899.238574:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434899.238578:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434899.238581:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873280:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434900.134221:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434901.158039:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434902.183140:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434903.206032:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434904.230299:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434904.357904:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434904.357914:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434904.357918:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434904.357929:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7273 00000100:00080000:2.0:1789434904.357938:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434904.357950:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434904.357986:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913873408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434904.358011:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434904.358015:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434904.358017:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434904.358020:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7273 00000100:00080000:2.0:1789434904.358025:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434904.358030:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434904.358039:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434904.358051:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913873536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434904.358060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434904.358063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434904.358065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434904.358068:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7273 00000100:00080000:2.0:1789434904.358071:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434904.358075:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434904.358080:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434904.358091:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913873664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434904.358098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913873408:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434904.358105:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434904.358109:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434904.358110:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434904.358115:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7273 00000100:00080000:2.0:1789434904.358118:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434904.358123:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434904.358128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434904.358137:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913873792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789434904.358201:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434904.358240:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434904/real 1789434904] req@ffff9811d2101500 x1876350913873408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434904.358268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913873408/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434904.358279:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434904.358283:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434904.358287:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913873408:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434904.358293:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434904.358312:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434904.358323:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434904/real 1789434904] req@ffff9811d2101880 x1876350913873536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434904.358334:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913873536/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434904.358342:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434904.358345:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434904.358348:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434904.358368:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873664:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434904.358390:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434904.358406:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434904/real 1789434904] req@ffff9811d2103b80 x1876350913873664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434904.358417:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913873664/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434904.358424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434904.358427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434904.358430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873664:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434904.358436:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873792:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434904.358452:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434904.358460:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434904/real 1789434904] req@ffff9811d2101c00 x1876350913873792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434904.358470:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913873792/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434904.358479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434904.358482:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434904.358485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913873792:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434905.254083:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434906.278013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434907.301969:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434908.326031:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434909.350123:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789434909.477983:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434909.478011:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789434909.478015:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434909.478025:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7279 00000100:00080000:1.0:1789434909.478055:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434909.478066:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789434909.478103:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913873920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434909.478132:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434909.478135:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789434909.478137:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434909.478141:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7279 00000100:00080000:1.0:1789434909.478145:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434909.478149:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434909.478159:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434909.478170:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913874048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434909.478179:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434909.478182:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789434909.478184:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434909.478187:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7279 00000100:00080000:1.0:1789434909.478190:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434909.478194:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434909.478198:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434909.478209:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913874176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789434909.479423:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789434909.479439:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789434909.479441:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789434909.479447:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7279 00000100:00080000:1.0:1789434909.479453:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789434909.479460:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789434909.479468:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434909.479485:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913874304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434909.479544:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913873920:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434909.479625:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434909.479671:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434909/real 1789434909] req@ffff9811d0bdc700 x1876350913873920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434909.479712:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913873920/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434909.479722:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434909.479726:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434909.479729:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913873920:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434909.479736:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434909.479760:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434909.479772:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434909/real 1789434909] req@ffff9811d0bdc000 x1876350913874048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434909.479790:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913874048/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434909.479801:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434909.479804:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434909.479808:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434909.479813:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874176:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434909.479830:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434909.479856:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434909/real 1789434909] req@ffff9811d0bdd880 x1876350913874176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434909.479867:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913874176/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434909.479874:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434909.479877:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434909.479880:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874176:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434909.479885:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874304:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434909.479902:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434909.479911:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434909/real 1789434909] req@ffff9811d0bde680 x1876350913874304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434909.479926:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913874304/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434909.479940:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434909.479943:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434909.479946:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874304:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434910.374018:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434911.399532:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434912.422724:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434913.446186:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434914.470074:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434914.598169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434914.598183:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434914.598187:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434914.598196:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7284 00000100:00080000:2.0:1789434914.598204:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434914.598216:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434914.598250:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913874432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434914.598283:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434914.598287:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434914.598288:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434914.598292:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7284 00000100:00080000:2.0:1789434914.598295:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434914.598299:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434914.598315:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434914.598403:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913874432:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434914.598481:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434914.598531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434914/real 1789434914] req@ffff9811d2100000 x1876350913874432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434914.598548:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913874432/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434914.598559:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434914.598563:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434914.598566:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913874432:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789434914.598626:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913874560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434914.598640:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434914.598643:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434914.598644:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434914.598647:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7284 00000100:00080000:2.0:1789434914.598650:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434914.598667:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434914.598673:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434914.598685:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913874688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434914.598699:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434914.598702:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434914.598703:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434914.598707:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7284 00000100:00080000:2.0:1789434914.598711:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434914.598716:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434914.598721:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434914.598733:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913874816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434914.598851:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434914.598876:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434914.598948:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434914/real 1789434914] req@ffff9811d2103800 x1876350913874560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434914.598962:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913874560/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434914.598970:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434914.598973:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434914.598976:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434914.598983:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874688:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434914.599004:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434914.599015:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434914/real 1789434914] req@ffff9811d2102d80 x1876350913874688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434914.599025:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913874688/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434914.599031:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434914.599034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434914.599037:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874688:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434914.599043:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874816:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434914.599062:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434914.599072:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434914/real 1789434914] req@ffff9811d2100000 x1876350913874816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434914.599082:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913874816/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434914.599090:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434914.599093:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434914.599096:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913874816:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434915.494018:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434916.518138:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434917.542037:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434918.569252:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434919.590772:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434919.717999:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434919.718009:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434919.718013:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434919.718023:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7289 00000100:00080000:3.0:1789434919.718031:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434919.718042:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434919.718074:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913874944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434919.718182:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913874944:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434919.718234:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434919.718260:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434919.718264:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434919.718266:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434919.718270:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7289 00000100:00080000:3.0:1789434919.718273:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434919.718278:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434919.718287:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434919.718299:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913875072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434919.718380:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434919/real 1789434919] req@ffff9811d1bc7480 x1876350913874944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434919.718397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913874944/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434919.718407:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434919.718410:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434919.718414:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913874944:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434919.718432:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434919.718451:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434919.718461:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434919/real 1789434919] req@ffff9811d1bc6a00 x1876350913875072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434919.718472:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913875072/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434919.718480:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434919.718483:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434919.718486:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875072:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789434919.722908:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434919.722916:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434919.722920:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434919.722928:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7289 00000100:00080000:3.0:1789434919.722936:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434919.722945:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434919.722956:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434919.722983:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913875200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434919.723012:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434919.723016:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434919.723018:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434919.723022:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7289 00000100:00080000:3.0:1789434919.723026:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434919.723030:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434919.723035:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434919.723047:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913875328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434919.723120:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875200:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434919.723172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434919.723198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434919/real 1789434919] req@ffff9811d1bc5180 x1876350913875200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434919.723219:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913875200/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434919.723229:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434919.723232:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434919.723236:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875200:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434919.723243:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875328:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434919.723264:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434919.723275:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434919/real 1789434919] req@ffff9811d1bc7800 x1876350913875328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434919.723286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913875328/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434919.723295:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434919.723298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434919.723301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875328:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434920.616209:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434921.638664:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434922.662408:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434923.686041:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434924.711630:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434924.837969:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434924.837983:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434924.837988:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434924.838000:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7294 00000100:00080000:3.0:1789434924.838012:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434924.838027:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434924.838074:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913875456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434924.838110:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434924.838114:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434924.838116:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434924.838121:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7294 00000100:00080000:3.0:1789434924.838125:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434924.838133:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434924.838146:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434924.838162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913875456:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434924.838164:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913875584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434924.838183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434924.838187:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434924.838190:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434924.838195:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7294 00000100:00080000:3.0:1789434924.838200:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434924.838219:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434924.838225:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434924.838240:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913875712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434924.838251:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434924.838255:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434924.838257:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434924.838262:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7294 00000100:00080000:3.0:1789434924.838267:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434924.838273:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434924.838278:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434924.838280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789434924.838296:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913875840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434924.838313:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875712:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434924.838331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875840:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434925.734447:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434926.758576:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434927.782119:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789434927.913937:0:2379:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789434927.913983:0:2379:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789434927.913999:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434927.914595:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434924/real 1789434927] req@ffff9811d1bc5880 x1876350913875456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434949 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434927.914619:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913875456/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434949 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434927.914630:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434927.914635:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434927.914638:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913875456:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434927.914875:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434927.914890:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789434927.914897:0:2379:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:0.0:1789434927.915006:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434924/real 1789434927] req@ffff9811d1bc4000 x1876350913875840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434927.915022:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913875840/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434927.915031:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434927.915034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434927.915038:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875840:192.168.203.128@tcp:8:kworker.0 00000100:00100000:0.0:1789434927.915044:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434924/real 1789434927] req@ffff9811d1bc6680 x1876350913875584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434927.915055:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913875584/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434927.915062:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434927.915065:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434927.915068:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434927.915074:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434924/real 1789434927] req@ffff9811d1bc5c00 x1876350913875712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434927.915084:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913875712/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434927.915091:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434927.915094:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434927.915097:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913875712:192.168.203.128@tcp:38:kworker.0 00010000:00010000:3.0:1789434928.806393:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434929.830752:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434929.957990:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434929.957999:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434929.958003:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434929.958017:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7299 00000100:00080000:2.0:1789434929.958025:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434929.958036:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434929.958066:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913875968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434929.958093:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434929.958097:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434929.958098:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434929.958102:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7299 00000100:00080000:2.0:1789434929.958106:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434929.958110:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434929.958119:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434929.958131:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913876096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434929.958140:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434929.958155:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434929.958157:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434929.958160:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7299 00000100:00080000:2.0:1789434929.958163:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434929.958168:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434929.958172:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434929.958183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913876224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434929.958191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434929.958194:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434929.958196:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434929.958199:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7299 00000100:00080000:2.0:1789434929.958202:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434929.958206:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434929.958209:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434929.958218:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913876352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434929.958274:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913875968:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434929.958327:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434929.958367:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434929/real 1789434929] req@ffff9811d2101c00 x1876350913875968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434929.958384:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913875968/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434929.958395:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434929.958399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434929.958402:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913875968:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434929.958409:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434929.958430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434929.958441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434929/real 1789434929] req@ffff9811d2103b80 x1876350913876096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434929.958452:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913876096/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434929.958459:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434929.958462:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434929.958465:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434929.958471:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876224:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434929.958485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434929.958493:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434929/real 1789434929] req@ffff9811d2101880 x1876350913876224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434929.958504:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913876224/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434929.958512:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434929.958514:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434929.958517:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876224:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434929.958522:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876352:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434929.958536:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434929.958558:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434929/real 1789434929] req@ffff9811d2101500 x1876350913876352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434929.958568:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913876352/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434929.958575:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434929.958578:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434929.958582:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876352:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434930.855861:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434931.878050:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434932.902051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434933.926019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434934.950195:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434935.077961:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434935.077970:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434935.077974:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434935.077984:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7304 00000100:00080000:2.0:1789434935.077996:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434935.078007:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434935.078042:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913876480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434935.078067:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434935.078070:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434935.078072:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434935.078075:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7304 00000100:00080000:2.0:1789434935.078078:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434935.078082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434935.078091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434935.078146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913876608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434935.078155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434935.078158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434935.078160:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434935.078164:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7304 00000100:00080000:2.0:1789434935.078167:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434935.078171:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434935.078175:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434935.078185:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913876736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434935.078194:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434935.078196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434935.078198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434935.078201:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7304 00000100:00080000:2.0:1789434935.078204:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434935.078208:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434935.078211:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434935.078219:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913876864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789434935.078572:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913876480:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789434935.078637:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434935.078780:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434935/real 1789434935] req@ffff9811d2102a00 x1876350913876480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434935.078810:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913876480/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434935.078825:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434935.078829:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434935.078863:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913876480:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789434935.078871:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434935.079433:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434935.079449:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434935/real 1789434935] req@ffff9811d2100e00 x1876350913876608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434935.079463:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913876608/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434935.079472:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434935.079475:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434935.079479:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434935.079486:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876736:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434935.079519:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434935.079582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434935/real 1789434935] req@ffff9811d2101f80 x1876350913876736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434935.079595:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913876736/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434935.079603:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434935.079606:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434935.079610:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876736:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789434935.079617:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876864:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789434935.079638:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789434935.079817:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434935/real 1789434935] req@ffff9811d2101180 x1876350913876864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789434935.079857:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913876864/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789434935.079866:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789434935.079870:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789434935.079873:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913876864:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434935.974221:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434936.998013:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434938.022216:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434939.045997:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434940.070065:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434940.198135:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434940.198154:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434940.198159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434940.198171:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7309 00000100:00080000:0.0:1789434940.198181:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434940.198193:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434940.198250:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913876992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434940.198282:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434940.198285:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434940.198287:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434940.198290:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7309 00000100:00080000:0.0:1789434940.198293:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434940.198298:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434940.198306:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434940.198316:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913877120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434940.198322:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434940.198325:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434940.198326:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434940.198329:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7309 00000100:00080000:0.0:1789434940.198331:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434940.198333:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434940.198336:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434940.198345:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913877248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434940.198355:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913876992:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789434940.198359:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434940.198377:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434940.198379:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434940.198384:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7309 00000100:00080000:0.0:1789434940.198388:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434940.198394:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434940.198399:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434940.198411:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913877376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434940.198442:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434940.198583:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434940/real 1789434940] req@ffff9811c8cd5180 x1876350913876992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434940.198602:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913876992/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434940.198610:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434940.198613:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434940.198616:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913876992:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434940.198625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434940.198663:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434940.198672:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434940/real 1789434940] req@ffff9811c8cd7800 x1876350913877120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434940.198678:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913877120/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434940.198683:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434940.198685:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434940.198687:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434940.198709:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877248:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434940.198727:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434940.198737:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434940/real 1789434940] req@ffff9811c8cd7480 x1876350913877248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434940.198749:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913877248/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434940.198757:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434940.198760:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434940.198763:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877248:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434940.198769:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877376:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434940.198790:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434940.198801:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434940/real 1789434940] req@ffff9811c8cd6a00 x1876350913877376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434940.198812:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913877376/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434940.198819:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434940.198822:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434940.198825:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877376:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434941.094384:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434942.118827:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434943.142017:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434944.166014:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434945.190017:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434945.318002:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434945.318013:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434945.318018:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434945.318029:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7314 00000100:00080000:3.0:1789434945.318038:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434945.318050:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434945.318097:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913877504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434945.318122:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434945.318126:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434945.318129:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434945.318133:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7314 00000100:00080000:3.0:1789434945.318137:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434945.318142:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434945.318152:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434945.318167:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913877632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434945.318183:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434945.318186:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434945.318187:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434945.318191:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7314 00000100:00080000:3.0:1789434945.318193:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434945.318198:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434945.318202:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434945.318211:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913877760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434945.318221:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434945.318224:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434945.318226:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434945.318233:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7314 00000100:00100000:1.0:1789434945.318235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913877504:192.168.203.128@tcp:250:kworker.0 00000100:00080000:3.0:1789434945.318237:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434945.318244:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434945.318249:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434945.318262:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913877888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434945.318301:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434945.318339:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434945/real 1789434945] req@ffff9811d1bc5f80 x1876350913877504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434945.318366:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913877504/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434945.318380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434945.318384:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434945.318388:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913877504:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434945.318395:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434945.318418:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434945.318443:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434945/real 1789434945] req@ffff9811d1bc7100 x1876350913877632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434945.318455:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913877632/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434945.318464:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434945.318468:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434945.318472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434945.318479:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877760:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434945.318500:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434945.318511:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434945/real 1789434945] req@ffff9811d1bc5500 x1876350913877760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434945.318521:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913877760/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434945.318535:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434945.318538:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434945.318541:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877760:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434945.318565:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877888:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434945.318585:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434945.318596:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434945/real 1789434945] req@ffff9811d1bc6300 x1876350913877888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434945.318608:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913877888/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434945.318616:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434945.318619:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434945.318622:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913877888:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434946.214070:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434947.239296:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434948.262014:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434949.287620:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434950.310110:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434950.438066:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434950.438076:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434950.438079:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434950.438096:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7320 00000100:00080000:2.0:1789434950.438107:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434950.438119:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434950.438165:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913878016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434950.438192:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434950.438196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434950.438198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434950.438202:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7320 00000100:00080000:2.0:1789434950.438205:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434950.438210:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434950.438219:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434950.438230:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913878144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434950.438239:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434950.438242:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434950.438244:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434950.438247:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7320 00000100:00080000:2.0:1789434950.438250:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434950.438254:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434950.438259:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434950.438267:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913878272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434950.438276:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434950.438279:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434950.438281:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434950.438284:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7320 00000100:00080000:2.0:1789434950.438286:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434950.438290:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434950.438294:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434950.438304:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913878400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434950.439273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913878016:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434950.439344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434950.439383:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434950/real 1789434950] req@ffff9811d2101180 x1876350913878016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434950.439413:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913878016/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434950.439424:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434950.439428:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434950.439440:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913878016:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434950.439448:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434950.439470:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434950.439484:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434950/real 1789434950] req@ffff9811d2101f80 x1876350913878144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434950.439496:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913878144/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434950.439503:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434950.439506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434950.439509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434950.439514:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878272:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434950.439532:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434950.439542:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434950/real 1789434950] req@ffff9811d2100e00 x1876350913878272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434950.439552:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913878272/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434950.439559:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434950.439562:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434950.439565:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878272:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434950.439570:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878400:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434950.439588:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434950.439598:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434950/real 1789434950] req@ffff9811d2102a00 x1876350913878400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434950.439610:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913878400/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434950.439617:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434950.439620:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434950.439623:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878400:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434951.334319:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434952.359087:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434953.382055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434954.406058:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434955.430023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434955.557942:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434955.557953:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434955.557957:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434955.557968:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7325 00000100:00080000:2.0:1789434955.557977:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434955.557988:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434955.558026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913878528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434955.558064:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434955.558069:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434955.558071:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434955.558076:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7325 00000100:00080000:2.0:1789434955.558079:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434955.558084:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434955.558093:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434955.558108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913878656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434955.558113:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913878528:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434955.558124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434955.558127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434955.558129:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434955.558133:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7325 00000100:00080000:2.0:1789434955.558137:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434955.558141:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434955.558147:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434955.558159:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913878784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434955.558169:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434955.558172:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434955.558174:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789434955.558179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434955.558180:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7325 00000100:00080000:2.0:1789434955.558186:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434955.558192:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434955.558198:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434955.558210:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913878912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434955.558221:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434955/real 1789434955] req@ffff9811d2101500 x1876350913878528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434955.558240:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913878528/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434955.558249:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434955.558252:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434955.558256:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913878528:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434955.558275:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434955.558296:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434955.558309:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434955/real 1789434955] req@ffff9811d2101880 x1876350913878656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434955.558321:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913878656/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434955.558334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434955.558338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434955.558341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434955.558347:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878784:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434955.558376:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434955.558386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434955/real 1789434955] req@ffff9811d2103b80 x1876350913878784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434955.558397:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913878784/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434955.558404:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434955.558408:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434955.558411:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878784:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434955.558416:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878912:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434955.558431:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434955.558440:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434955/real 1789434955] req@ffff9811d2101c00 x1876350913878912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434955.558451:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913878912/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434955.558458:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434955.558461:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434955.558464:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913878912:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434956.454017:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434957.478526:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434958.502185:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434959.526005:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434960.550171:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434960.678306:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434960.678330:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434960.678334:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434960.678344:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7330 00000100:00080000:2.0:1789434960.678354:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434960.678366:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434960.678401:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913879040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434960.678433:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434960.678437:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434960.678439:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434960.678443:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7330 00000100:00080000:2.0:1789434960.678447:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434960.678452:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434960.678461:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434960.678473:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913879168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434960.678483:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434960.678486:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434960.678488:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434960.678491:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7330 00000100:00080000:2.0:1789434960.678496:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434960.678500:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434960.678504:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434960.678514:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913879296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434960.678534:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434960.678537:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434960.678538:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434960.678542:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7330 00000100:00080000:2.0:1789434960.678545:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434960.678549:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434960.678553:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434960.678564:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913879424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434960.681694:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913879040:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434960.681765:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434960.681805:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434960/real 1789434960] req@ffff9811d2100000 x1876350913879040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434960.681823:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913879040/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434960.681868:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434960.681874:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434960.681878:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913879040:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434960.681886:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434960.681907:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434960.681918:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434960/real 1789434960] req@ffff9811d2102d80 x1876350913879168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434960.681929:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913879168/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434960.681936:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434960.681939:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434960.681942:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434960.681948:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879296:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434960.681964:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434960.681973:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434960/real 1789434960] req@ffff9811d2103800 x1876350913879296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434960.681984:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913879296/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434960.681991:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434960.681994:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434960.681997:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879296:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434960.682002:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879424:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434960.682019:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434960.682028:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434960/real 1789434960] req@ffff9811d2101c00 x1876350913879424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434960.682039:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913879424/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434960.682046:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434960.682049:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434960.682052:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879424:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434961.574197:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434962.598088:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434963.622319:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434964.646099:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434965.670056:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434965.798031:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434965.798041:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434965.798045:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434965.798056:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7335 00000100:00080000:2.0:1789434965.798064:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434965.798074:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434965.798106:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913879552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434965.798139:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434965.798142:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434965.798149:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434965.798154:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7335 00000100:00080000:2.0:1789434965.798157:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434965.798162:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434965.798172:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434965.798189:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913879680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434965.798200:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434965.798203:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434965.798204:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434965.798210:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7335 00000100:00100000:1.0:1789434965.798214:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913879552:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434965.798215:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434965.798222:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434965.798227:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434965.798244:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913879808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434965.798254:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434965.798258:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434965.798260:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434965.798264:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7335 00000100:00080000:2.0:1789434965.798273:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434965.798290:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789434965.798295:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:2.0:1789434965.798298:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434965.798312:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913879936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434965.798340:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434965/real 1789434965] req@ffff9811d2103b80 x1876350913879552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434965.798369:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913879552/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434965.798380:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434965.798385:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434965.798389:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913879552:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434965.798411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879680:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434965.798447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434965.798459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434965/real 1789434965] req@ffff9811d2101880 x1876350913879680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434965.798469:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913879680/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434965.798477:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434965.798493:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434965.798497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434965.798503:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879808:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434965.798524:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434965.798536:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434965/real 1789434965] req@ffff9811d2101500 x1876350913879808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434965.798547:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913879808/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434965.798560:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434965.798564:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434965.798567:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879808:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434965.798574:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879936:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434965.798594:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434965.798603:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434965/real 1789434965] req@ffff9811d2102a00 x1876350913879936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434965.798614:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913879936/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434965.798621:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434965.798624:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434965.798627:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913879936:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434966.694058:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434967.718029:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434968.742176:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434969.766383:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434970.790288:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434970.918002:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434970.918012:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434970.918016:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434970.918026:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7340 00000100:00080000:3.0:1789434970.918034:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434970.918045:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434970.918079:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4700 x1876350913880064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434970.918105:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434970.918117:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434970.918119:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434970.918123:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7340 00000100:00080000:3.0:1789434970.918128:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434970.918134:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434970.918145:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434970.918158:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913880064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434970.918162:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6300 x1876350913880192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434970.918176:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434970.918179:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434970.918181:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434970.918184:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7340 00000100:00080000:3.0:1789434970.918187:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434970.918192:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434970.918197:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434970.918208:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5500 x1876350913880320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434970.918218:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434970.918221:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789434970.918223:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789434970.918230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434970.918249:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7340 00000100:00080000:3.0:1789434970.918255:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434970.918271:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434970.918278:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434970.918284:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434970/real 1789434970] req@ffff9811d1bc4700 x1876350913880064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789434970.918293:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7100 x1876350913880448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434970.918310:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4700 x1876350913880064/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434970.918320:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434970.918323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434970.918327:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913880064:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434970.918344:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434970.918366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434970.918378:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434970/real 1789434970] req@ffff9811d1bc6300 x1876350913880192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434970.918391:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6300 x1876350913880192/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434970.918399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434970.918402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434970.918406:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434970.918411:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880320:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434970.918430:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434970.918441:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434970/real 1789434970] req@ffff9811d1bc5500 x1876350913880320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434970.918453:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5500 x1876350913880320/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434970.918460:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434970.918463:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434970.918467:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880320:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434970.918473:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880448:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434970.918492:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434970.918502:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434970/real 1789434970] req@ffff9811d1bc7100 x1876350913880448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434970.918513:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7100 x1876350913880448/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789434995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434970.918525:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434970.918527:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434970.918530:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880448:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434971.814034:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434972.838274:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434973.862409:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434974.886020:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434975.913323:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789434976.037984:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434976.037995:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789434976.038000:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434976.038010:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7345 00000100:00080000:3.0:1789434976.038019:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434976.038031:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789434976.038075:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5f80 x1876350913880576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434976.038098:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434976.038101:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789434976.038103:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434976.038107:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7345 00000100:00080000:3.0:1789434976.038111:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434976.038116:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434976.038126:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789434976.038134:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913880576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789434976.038141:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5c00 x1876350913880704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434976.038153:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434976.038156:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789434976.038158:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434976.038161:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7345 00000100:00080000:3.0:1789434976.038165:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434976.038170:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434976.038175:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789434976.038187:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6680 x1876350913880832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789434976.038213:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789434976.038219:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1789434976.038220:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789434976.038222:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789434976.038228:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7345 00000100:00080000:3.0:1789434976.038232:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789434976.038238:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789434976.038244:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789434976.038257:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc4000 x1876350913880960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434976.038263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434976/real 1789434976] req@ffff9811d1bc5f80 x1876350913880576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434976.038283:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5f80 x1876350913880576/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434976.038294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434976.038298:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434976.038301:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913880576:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434976.038322:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434976.038344:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434976.038357:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434976/real 1789434976] req@ffff9811d1bc5c00 x1876350913880704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434976.038368:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5c00 x1876350913880704/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434976.038376:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434976.038379:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434976.038382:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434976.038388:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880832:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434976.038417:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434976.038427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434976/real 1789434976] req@ffff9811d1bc6680 x1876350913880832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434976.038629:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6680 x1876350913880832/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434976.038637:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434976.038640:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434976.038643:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880832:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434976.038649:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880960:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434976.038669:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434976.038679:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434976/real 1789434976] req@ffff9811d1bc4000 x1876350913880960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434976.038689:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc4000 x1876350913880960/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434976.038696:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434976.038699:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434976.038701:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913880960:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434976.934015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434977.958070:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434978.985630:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434980.006888:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434981.031539:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434981.158110:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434981.158120:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434981.158125:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434981.158137:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7350 00000100:00080000:2.0:1789434981.158148:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434981.158171:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434981.158214:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913881088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434981.158253:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434981.158257:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434981.158259:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434981.158264:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7350 00000100:00080000:2.0:1789434981.158267:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434981.158272:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434981.158282:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434981.158294:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913881216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434981.158304:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434981.158307:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434981.158309:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434981.158312:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7350 00000100:00080000:2.0:1789434981.158315:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434981.158319:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434981.158323:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434981.158333:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913881344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434981.158342:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434981.158346:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434981.158347:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434981.158351:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7350 00000100:00080000:2.0:1789434981.158354:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434981.158358:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434981.158361:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434981.158369:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913881472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434981.158527:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913881088:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434981.158605:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434981.158646:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434981/real 1789434981] req@ffff9811d2100e00 x1876350913881088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434981.158665:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913881088/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434981.158677:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434981.158681:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434981.158685:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913881088:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434981.158692:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434981.158710:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434981.158720:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434981/real 1789434981] req@ffff9811d2101f80 x1876350913881216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434981.158731:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913881216/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434981.158739:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434981.158742:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434981.158745:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434981.158750:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881344:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434981.158763:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434981.158771:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434981/real 1789434981] req@ffff9811d2101180 x1876350913881344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434981.158781:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913881344/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434981.158795:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434981.158798:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434981.158801:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881344:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434981.158807:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881472:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434981.158827:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434981.158862:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434981/real 1789434981] req@ffff9811d2102a00 x1876350913881472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434981.158874:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913881472/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434981.158882:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434981.158885:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434981.158889:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881472:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434982.054168:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434983.082470:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434984.102176:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434985.126256:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434986.150168:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434986.277981:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434986.277991:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434986.277996:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434986.278007:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7355 00000100:00080000:2.0:1789434986.278017:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434986.278028:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434986.278063:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913881600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434986.278438:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913881600:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789434986.278499:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434986.278531:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434986.278535:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434986.278537:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434986.278541:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7355 00000100:00080000:2.0:1789434986.278545:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434986.278551:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434986.278561:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434986.278574:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913881728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434986.279943:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434986/real 1789434986] req@ffff9811d2101500 x1876350913881600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434986.279968:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913881600/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434986.279979:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434986.279984:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434986.279988:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913881600:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434986.280017:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434986.280059:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434986.280074:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434986/real 1789434986] req@ffff9811d2101880 x1876350913881728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434986.280087:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913881728/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434986.280095:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434986.280099:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434986.280102:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881728:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789434986.280866:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434986.280873:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434986.280875:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434986.280881:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7355 00000100:00080000:2.0:1789434986.280888:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434986.280897:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434986.280906:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434986.280926:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913881856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434986.280950:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434986.280954:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434986.280956:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434986.280959:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7355 00000100:00080000:2.0:1789434986.280962:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434986.280967:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434986.280972:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789434986.280980:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789434986.280983:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913881984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789434986.281018:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434986.281033:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434986/real 1789434986] req@ffff9811d2103b80 x1876350913881856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434986.281045:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913881856/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434986.281055:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434986.281058:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434986.281061:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881856:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434986.281082:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881984:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434986.281104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434986.281114:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434986/real 1789434986] req@ffff9811d2101c00 x1876350913881984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434986.281138:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913881984/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434986.281147:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434986.281150:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434986.281153:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913881984:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434987.174026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434988.198077:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434989.222023:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434990.246155:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434991.270021:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789434991.397982:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434991.397992:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789434991.397996:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434991.398006:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7360 00000100:00080000:0.0:1789434991.398015:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434991.398027:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789434991.398061:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913882112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434991.398100:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434991.398104:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789434991.398106:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434991.398111:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7360 00000100:00080000:0.0:1789434991.398115:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434991.398119:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434991.398128:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434991.398139:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913882240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434991.398148:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434991.398152:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789434991.398153:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434991.398156:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7360 00000100:00080000:0.0:1789434991.398159:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434991.398163:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434991.398167:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789434991.398175:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913882368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789434991.398185:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789434991.398189:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789434991.398191:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789434991.398194:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7360 00000100:00080000:0.0:1789434991.398197:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789434991.398201:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789434991.398205:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789434991.398214:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913882496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434991.398331:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913882112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434991.398421:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434991.398446:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434991.398463:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434992.294037:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434993.320655:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434994.345185:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789434994.470083:0:2376:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789434994.470122:0:2376:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789434994.470136:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434994.470173:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434994.470263:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434991/real 1789434994] req@ffff9811c8cd4000 x1876350913882112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434994.470287:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913882112/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434994.470298:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434994.470302:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434994.470306:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913882112:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434994.470314:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434991/real 1789434994] req@ffff9811c8cd6300 x1876350913882240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434994.470326:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913882240/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434994.470335:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434994.470338:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434994.470341:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789434994.470381:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789434994.470402:0:2376:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789434994.470427:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434991/real 1789434994] req@ffff9811c8cd7b80 x1876350913882368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434994.470442:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913882368/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434994.470449:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434994.470453:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434994.470456:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882368:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434994.470463:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434991/real 1789434994] req@ffff9811c8cd4a80 x1876350913882496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434994.470474:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913882496/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434994.470482:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434994.470485:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434994.470488:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882496:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789434995.366373:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434996.390251:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789434996.518191:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434996.518202:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789434996.518206:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434996.518268:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7366 00000100:00080000:2.0:1789434996.518276:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434996.518287:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789434996.518321:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913882624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434996.518346:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434996.518349:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789434996.518351:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434996.518357:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7366 00000100:00080000:2.0:1789434996.518360:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434996.518365:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434996.518375:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434996.518388:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913882752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434996.518393:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913882624:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789434996.518406:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789434996.518410:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789434996.518412:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434996.518417:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7366 00000100:00080000:2.0:1789434996.518420:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434996.518424:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434996.518436:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789434996.518447:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913882880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789434996.518457:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789434996.518462:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789434996.518462:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789434996.518466:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789434996.518471:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7366 00000100:00080000:2.0:1789434996.518476:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789434996.518481:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789434996.518485:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789434996.518497:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913883008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789434996.518507:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434996/real 1789434996] req@ffff9811d2103800 x1876350913882624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434996.518527:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913882624/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434996.518538:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434996.518542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434996.518545:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913882624:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789434996.518568:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434996.518590:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434996.518603:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434996/real 1789434996] req@ffff9811d2102d80 x1876350913882752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434996.518891:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913882752/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434996.518899:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434996.518903:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434996.518906:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434996.518912:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882880:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789434996.518934:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434996.518946:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434996/real 1789434996] req@ffff9811d2100000 x1876350913882880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434996.518957:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913882880/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434996.518971:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434996.518974:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434996.518977:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913882880:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789434996.518983:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883008:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789434996.519007:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789434996.519018:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789434996/real 1789434996] req@ffff9811d2101c00 x1876350913883008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789434996.519029:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913883008/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789434996.519037:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789434996.519040:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789434996.519043:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883008:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789434997.414241:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789434998.438129:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789434999.462329:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435000.491561:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435001.510098:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435001.638811:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435001.638828:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435001.638849:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435001.638857:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7371 00000100:00080000:1.0:1789435001.638864:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435001.638872:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435001.638902:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913883136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435001.638927:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435001.638930:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435001.638932:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435001.638936:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7371 00000100:00080000:1.0:1789435001.638939:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435001.638944:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435001.638953:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435001.638965:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913883264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435001.638985:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435001.638989:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435001.638990:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435001.638995:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7371 00000100:00080000:1.0:1789435001.638999:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435001.639003:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435001.639007:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435001.639028:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913883136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435001.639035:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913883392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435001.639046:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435001.639050:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435001.639051:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435001.639055:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7371 00000100:00080000:1.0:1789435001.639059:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435001.639063:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435001.639069:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435001.639080:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913883520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789435001.639092:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435001.639130:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435001/real 1789435001] req@ffff9811d0bdf480 x1876350913883136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435001.639142:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913883136/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435001.639150:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435001.639153:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435001.639156:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913883136:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435001.639163:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435001.639183:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435001.639195:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435001/real 1789435001] req@ffff9811d0bdfb80 x1876350913883264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435001.639207:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913883264/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435001.639212:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435001.639214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435001.639217:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435001.639233:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883392:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435001.639246:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435001.639252:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435001/real 1789435001] req@ffff9811d0bded80 x1876350913883392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435001.639259:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913883392/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435001.639263:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435001.639265:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435001.639267:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883392:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435001.639270:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883520:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435001.639280:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435001.639285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435001/real 1789435001] req@ffff9811d0bdc380 x1876350913883520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435001.639291:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913883520/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435001.639296:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435001.639297:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435001.639303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883520:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435002.534051:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435003.558039:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435004.582040:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435005.606055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435006.630092:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435006.757946:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435006.757957:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435006.757961:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435006.757970:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7376 00000100:00080000:2.0:1789435006.757990:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435006.758003:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435006.758047:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913883648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435006.758137:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435006.758141:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435006.758143:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435006.758148:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7376 00000100:00080000:2.0:1789435006.758152:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435006.758157:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435006.758168:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435006.758181:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913883776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435006.758191:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435006.758193:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435006.758195:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789435006.758197:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913883648:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789435006.758199:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7376 00000100:00080000:2.0:1789435006.758202:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435006.758205:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435006.758210:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435006.758219:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913883904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435006.758227:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435006.758230:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435006.758231:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435006.758234:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7376 00000100:00080000:2.0:1789435006.758236:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435006.758241:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435006.758244:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789435006.758252:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913884032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789435006.758267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435006.758303:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435006/real 1789435006] req@ffff9811d2103b80 x1876350913883648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435006.758316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913883648/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435006.758323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435006.758326:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435006.758328:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913883648:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435006.758345:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435006.758363:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435006.758374:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435006/real 1789435006] req@ffff9811d2101880 x1876350913883776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435006.758392:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913883776/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435006.758399:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435006.758402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435006.758405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435006.758410:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883904:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435006.758428:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435006.758437:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435006/real 1789435006] req@ffff9811d2101500 x1876350913883904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435006.758461:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913883904/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435006.758468:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435006.758471:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435006.758474:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913883904:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435006.758543:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884032:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435006.758586:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435006.758598:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435006/real 1789435006] req@ffff9811d2102a00 x1876350913884032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435006.758608:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913884032/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435006.758621:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435006.758636:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435006.758640:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884032:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435007.654199:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435008.678859:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435009.702007:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435010.726049:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435011.750057:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789435011.878147:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435011.878159:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789435011.878163:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435011.878207:0:148681:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7381 00000100:00080000:3.0:1789435011.878219:0:148681:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435011.878230:0:148681:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789435011.878264:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5880 x1876350913884160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789435011.878297:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435011.878301:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789435011.878304:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435011.878308:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7381 00000100:00080000:3.0:1789435011.878312:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435011.878328:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435011.878337:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435011.878350:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913884288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789435011.878359:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435011.878362:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789435011.878364:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435011.878367:0:148681:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7381 00000100:00080000:3.0:1789435011.878371:0:148681:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435011.878375:0:148681:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435011.878379:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435011.878388:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913884416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789435011.878397:0:148681:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435011.878400:0:148681:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789435011.878401:0:148681:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435011.878405:0:148681:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7381 00000100:00080000:3.0:1789435011.878408:0:148681:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435011.878412:0:148681:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435011.878416:0:148681:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789435011.878428:0:148681:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913884544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435011.879004:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913884160:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789435011.879067:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435011.879109:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435011/real 1789435011] req@ffff9811d1bc5880 x1876350913884160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435011.879124:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5880 x1876350913884160/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435011.879134:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435011.879138:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435011.879142:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913884160:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435011.879148:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435011.879168:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435011.879179:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435011/real 1789435011] req@ffff9811d1bc7800 x1876350913884288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435011.879189:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913884288/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435011.879195:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435011.879198:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435011.879201:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435011.879206:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884416:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435011.879222:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435011.879229:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435011/real 1789435011] req@ffff9811d1bc5180 x1876350913884416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435011.879239:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913884416/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435011.879245:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435011.879248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435011.879251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884416:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435011.879256:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884544:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435011.879269:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435011.879277:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435011/real 1789435011] req@ffff9811d1bc6a00 x1876350913884544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435011.879286:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913884544/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435011.879293:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435011.879295:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435011.879298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884544:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435012.774483:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435013.799384:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435014.823398:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435015.846045:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435016.871365:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435016.998080:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435016.998089:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435016.998094:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435016.998104:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7386 00000100:00080000:0.0:1789435016.998114:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435016.998125:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435016.998166:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913884672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435016.998232:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913884672:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435016.998290:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789435016.998323:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435016.998327:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435016.998338:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435016.998343:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7386 00000100:00080000:0.0:1789435016.998347:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435016.998353:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435016.998362:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435016.998377:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913884800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435016.999612:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435016/real 1789435016] req@ffff9811c8cd5500 x1876350913884672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435016.999632:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913884672/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435016.999642:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435016.999646:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435016.999649:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913884672:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435016.999674:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435016.999710:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435016.999725:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435016/real 1789435016] req@ffff9811c8cd6d80 x1876350913884800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435016.999738:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913884800/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435016.999746:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435016.999749:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435016.999752:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884800:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789435017.002225:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435017.002233:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435017.002237:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435017.002245:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7386 00000100:00080000:0.0:1789435017.002254:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435017.002265:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435017.002277:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435017.002306:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913884928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435017.002327:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435017.002341:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435017.002343:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435017.002347:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7386 00000100:00080000:0.0:1789435017.002350:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435017.002354:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435017.002359:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435017.002371:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913885056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435017.002625:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884928:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435017.002676:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435017.002701:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435017/real 1789435017] req@ffff9811c8cd5f80 x1876350913884928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435017.002716:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913884928/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435017.002725:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435017.002728:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435017.002732:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913884928:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435017.002738:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885056:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435017.002758:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435017.002770:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435017/real 1789435017] req@ffff9811c8cd6d80 x1876350913885056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435017.002780:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913885056/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435017.002788:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435017.002791:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435017.002794:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885056:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435017.894043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435018.918099:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435019.942079:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435020.968892:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435021.990264:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435022.119726:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435022.119734:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435022.119738:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435022.119748:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7391 00000100:00080000:1.0:1789435022.119759:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435022.119772:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435022.119807:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913885184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435022.119846:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435022.119849:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435022.119851:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435022.119856:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7391 00000100:00080000:1.0:1789435022.119878:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789435022.119882:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913885184:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789435022.119886:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435022.119899:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435022.119915:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913885312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435022.119925:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435022.119928:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435022.119930:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435022.119934:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7391 00000100:00080000:1.0:1789435022.119938:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435022.119942:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:0.0:1789435022.119947:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:1.0:1789435022.119950:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435022.119962:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913885440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435022.119971:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435022.119974:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435022.119976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435022.119980:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7391 00000100:00080000:1.0:1789435022.119983:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435022.119987:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435022.119992:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435022.119996:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435022/real 1789435022] req@ffff9811d0bdd180 x1876350913885184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435022.120005:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913885568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435022.120019:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913885184/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435022.120030:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435022.120034:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435022.120038:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913885184:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435022.120059:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435022.120080:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435022.120092:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435022/real 1789435022] req@ffff9811d0bddc00 x1876350913885312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435022.120104:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913885312/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435022.120112:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435022.120115:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435022.120117:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435022.120122:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885440:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435022.120140:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435022.120149:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435022/real 1789435022] req@ffff9811d0bdd500 x1876350913885440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435022.120161:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913885440/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435022.120167:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435022.120171:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435022.120173:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885440:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435022.120178:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885568:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435022.120196:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435022.120205:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435022/real 1789435022] req@ffff9811d0bdc380 x1876350913885568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435022.120214:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913885568/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435022.120227:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435022.120229:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435022.120232:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885568:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435023.015654:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435024.038099:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435025.062991:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435026.087115:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435027.110294:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435027.239129:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435027.239142:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435027.239147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435027.239159:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7396 00000100:00080000:0.0:1789435027.239170:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435027.239183:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435027.239231:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913885696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435027.239266:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435027.239270:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435027.239272:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435027.239277:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7396 00000100:00080000:0.0:1789435027.239280:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435027.239285:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435027.239295:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435027.239307:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913885824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435027.239316:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435027.239319:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435027.239322:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435027.239325:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7396 00000100:00080000:0.0:1789435027.239328:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435027.239332:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435027.239336:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435027.239344:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913885952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435027.239354:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435027.239357:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435027.239359:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435027.239361:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7396 00000100:00080000:0.0:1789435027.239364:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435027.239367:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435027.239371:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435027.239381:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913886080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435027.239466:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913885696:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435027.239540:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435027.239585:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435027/real 1789435027] req@ffff9811c8cd5500 x1876350913885696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435027.239603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913885696/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435027.239614:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435027.239618:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435027.239622:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913885696:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435027.239629:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435027.239658:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435027.239671:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435027/real 1789435027] req@ffff9811c8cd4a80 x1876350913885824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435027.239682:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913885824/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435027.239696:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435027.239699:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435027.239703:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435027.239708:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885952:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435027.239727:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435027.239736:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435027/real 1789435027] req@ffff9811c8cd7b80 x1876350913885952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435027.239746:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913885952/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435027.239754:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435027.239757:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435027.239760:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913885952:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435027.239765:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886080:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435027.239796:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435027.239805:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435027/real 1789435027] req@ffff9811c8cd6300 x1876350913886080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435027.239816:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913886080/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435027.239855:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435027.239859:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435027.239863:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886080:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435028.135215:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435029.159738:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435030.184882:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435031.210639:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435032.230272:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789435032.359119:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435032.359129:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789435032.359133:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435032.359143:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7401 00000100:00080000:3.0:1789435032.359163:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435032.359176:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:3.0:1789435032.359211:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7480 x1876350913886208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435032.359273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913886208:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789435032.359332:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789435032.359362:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435032.359374:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789435032.359376:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435032.359380:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7401 00000100:00080000:3.0:1789435032.359384:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435032.359390:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435032.359400:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435032.359419:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc6a00 x1876350913886336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435032.359729:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435032/real 1789435032] req@ffff9811d1bc7480 x1876350913886208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435032.359747:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7480 x1876350913886208/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435032.359756:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435032.359760:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435032.359764:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913886208:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435032.359797:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886336:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435032.359821:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435032.359850:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435032/real 1789435032] req@ffff9811d1bc6a00 x1876350913886336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435032.359863:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc6a00 x1876350913886336/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435032.359870:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435032.359873:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435032.359877:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886336:192.168.203.128@tcp:38:kworker.0 00000100:00080000:3.0:1789435032.359977:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435032.359981:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1789435032.359983:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435032.359986:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7401 00000100:00080000:3.0:1789435032.359991:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435032.359996:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435032.360002:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435032.360014:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc5180 x1876350913886464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435032.360059:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886464:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435032.360267:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:3.0:1789435032.360288:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789435032.360293:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1789435032.360294:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789435032.360298:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7401 00000100:00080000:3.0:1789435032.360302:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789435032.360307:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:3.0:1789435032.360312:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789435032.360325:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d1bc7800 x1876350913886592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435032.360364:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435032/real 1789435032] req@ffff9811d1bc5180 x1876350913886464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435032.360377:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc5180 x1876350913886464/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435032.360385:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435032.360388:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435032.360391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886464:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435032.360402:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886592:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435032.360423:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435032.360434:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435032/real 1789435032] req@ffff9811d1bc7800 x1876350913886592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435032.360445:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d1bc7800 x1876350913886592/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435032.360454:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435032.360457:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435032.360460:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d1bc7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886592:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435033.254028:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435034.278953:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435035.302861:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435036.326043:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435037.350595:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435037.477941:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435037.477975:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435037.477979:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435037.477991:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7407 00000100:00080000:1.0:1789435037.478000:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435037.478011:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435037.478058:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913886720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435037.478089:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435037.478093:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435037.478095:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435037.478100:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7407 00000100:00080000:1.0:1789435037.478104:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435037.478109:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435037.478118:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435037.478132:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913886848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435037.478141:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435037.478145:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435037.478146:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435037.478149:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7407 00000100:00080000:1.0:1789435037.478152:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435037.478157:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435037.478161:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435037.478171:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913886976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435037.478181:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435037.478184:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435037.478186:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435037.478189:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7407 00000100:00080000:1.0:1789435037.478192:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435037.478197:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435037.478201:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435037.478211:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913887104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435037.478693:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913886720:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789435037.478858:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435037.479165:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435037/real 1789435037] req@ffff9811d0bded80 x1876350913886720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435037.479186:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913886720/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435037.479196:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435037.479200:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435037.479204:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913886720:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435037.479210:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886848:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435037.479234:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435037.479247:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435037/real 1789435037] req@ffff9811d0bdfb80 x1876350913886848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435037.479265:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913886848/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435037.479273:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435037.479276:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435037.479279:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886848:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435037.479285:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886976:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435037.479305:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435037.479317:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435037/real 1789435037] req@ffff9811d0bdf480 x1876350913886976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435037.479328:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913886976/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435037.479336:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435037.479339:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435037.479342:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913886976:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435037.479348:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887104:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435037.479366:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435037.479375:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435037/real 1789435037] req@ffff9811d0bde680 x1876350913887104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435037.479385:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913887104/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435037.479398:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435037.479402:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435037.479405:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887104:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435038.374285:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435039.398055:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435040.422059:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435041.446002:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435042.472033:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435042.598354:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435042.598364:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435042.598369:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435042.598380:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7412 00000100:00080000:1.0:1789435042.598389:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435042.598402:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435042.598437:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913887232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435042.598515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913887232:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789435042.598574:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789435042.598608:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435042.598613:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435042.598616:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435042.598620:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7412 00000100:00080000:1.0:1789435042.598624:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435042.598630:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435042.598639:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435042.598653:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913887360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435042.599352:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435042/real 1789435042] req@ffff9811d0bdd880 x1876350913887232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435042.599525:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913887232/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435042.599537:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435042.599542:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435042.599546:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913887232:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435042.599569:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887360:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435042.599599:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435042.599612:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435042/real 1789435042] req@ffff9811d0bdc000 x1876350913887360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435042.599672:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435042.599677:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435042.599679:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435042.599684:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7412 00000100:00080000:1.0:1789435042.599689:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435042.599696:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435042.599703:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435042.599791:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913887360/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435042.599803:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435042.599807:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435042.599811:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887360:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435042.599815:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913887488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435042.599855:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435042.599860:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1789435042.599861:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887488:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789435042.599864:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435042.599871:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7412 00000100:00080000:1.0:1789435042.599876:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789435042.599891:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:1.0:1789435042.599892:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435042.599900:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435042.599909:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435042/real 1789435042] req@ffff9811d0bdc700 x1876350913887488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435042.599916:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913887616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435042.599926:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913887488/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435042.599937:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435042.599941:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435042.599945:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887488:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435042.599958:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887616:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435042.599977:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435042.599988:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435042/real 1789435042] req@ffff9811d0bdea00 x1876350913887616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435042.599999:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913887616/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435042.600007:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435042.600010:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435042.600013:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887616:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435043.494931:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435044.518234:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435045.542293:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435046.566141:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435047.590325:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435047.718306:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435047.718320:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435047.718326:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435047.718339:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7417 00000100:00080000:0.0:1789435047.718349:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435047.718363:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435047.718422:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913887744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435047.718469:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435047.718473:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435047.718475:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435047.718480:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7417 00000100:00080000:0.0:1789435047.718484:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435047.718489:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435047.718760:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435047.718779:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913887872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435047.718791:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435047.718794:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435047.718796:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435047.718800:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7417 00000100:00080000:0.0:1789435047.718804:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435047.718809:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435047.718814:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435047.718822:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913888000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435047.718872:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435047.718876:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435047.718878:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435047.718882:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7417 00000100:00080000:0.0:1789435047.718886:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435047.718891:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435047.718896:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435047.718908:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913888128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435047.721804:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913887744:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435047.721963:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435047.722037:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435047/real 1789435047] req@ffff9811c8cd4000 x1876350913887744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435047.722062:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913887744/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435047.722076:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435047.722082:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435047.722087:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913887744:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435047.722096:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887872:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435047.722121:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435047.722134:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435047/real 1789435047] req@ffff9811c8cd6a00 x1876350913887872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435047.722145:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913887872/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435047.722152:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435047.722156:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435047.722159:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913887872:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435047.722166:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888000:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435047.722188:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435047.722198:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435047/real 1789435047] req@ffff9811c8cd7480 x1876350913888000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435047.722208:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913888000/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435047.722215:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435047.722219:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435047.722221:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888000:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435047.722229:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888128:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435047.722249:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435047.722258:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435047/real 1789435047] req@ffff9811c8cd7800 x1876350913888128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435047.722268:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913888128/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435047.722277:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435047.722280:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435047.722282:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888128:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435048.614407:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435049.638139:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435050.662711:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435051.686132:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435052.711129:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435052.837962:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435052.837972:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435052.837976:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435052.837987:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7422 00000100:00080000:0.0:1789435052.837996:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435052.838008:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435052.838041:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913888256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435052.838071:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435052.838075:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435052.838076:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435052.838080:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7422 00000100:00080000:0.0:1789435052.838084:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435052.838088:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435052.838107:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435052.838121:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913888384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435052.838130:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435052.838133:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435052.838134:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435052.838138:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7422 00000100:00080000:0.0:1789435052.838141:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435052.838145:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435052.838150:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435052.838159:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913888512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435052.838178:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435052.838182:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435052.838183:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435052.838187:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7422 00000100:00080000:0.0:1789435052.838190:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435052.838194:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435052.838199:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435052.838209:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913888640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435052.838392:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913888256:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435052.838449:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435052.838487:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435052/real 1789435052] req@ffff9811c8cd5180 x1876350913888256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435052.838509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913888256/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435052.838520:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435052.838524:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435052.838527:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913888256:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435052.838534:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888384:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435052.838554:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435052.838565:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435052/real 1789435052] req@ffff9811c8cd7100 x1876350913888384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435052.838576:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913888384/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435052.838583:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435052.838586:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435052.838589:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888384:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435052.838594:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888512:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435052.838612:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435052.838621:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435052/real 1789435052] req@ffff9811c8cd5c00 x1876350913888512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435052.838646:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913888512/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435052.838775:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435052.838779:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435052.838783:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888512:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435052.838789:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888640:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435052.838812:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435052.838823:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435052/real 1789435052] req@ffff9811c8cd4700 x1876350913888640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435052.838862:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913888640/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435052.838871:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435052.838874:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435052.838877:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888640:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435053.734053:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435054.759457:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435055.782338:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435056.806086:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435057.830029:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435057.958081:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435057.958097:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435057.958103:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435057.958117:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7427 00000100:00080000:0.0:1789435057.958130:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435057.958146:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435057.958200:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913888768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435057.958303:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913888768:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789435057.958939:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435057.958945:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435057.958948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435057.958955:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7427 00000100:00080000:0.0:1789435057.958960:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435057.958967:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435057.958978:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435057.958996:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913888896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435057.963637:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888896:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789435057.965860:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435057.965873:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435057.965878:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435057.965897:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7427 00000100:00080000:0.0:1789435057.965909:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435057.965922:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435057.965939:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435057.965982:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913889024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435057.966005:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435057.966009:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435057.966011:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435057.966015:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7427 00000100:00080000:0.0:1789435057.966018:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435057.966022:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435057.966027:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435057.966038:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913889152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435057.967107:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435057.967194:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889152:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435058.854223:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435059.879172:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435060.902314:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789435061.032256:0:2377:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:0.0:1789435061.032305:0:2377:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789435061.032321:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789435061.036330:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435057/real 1789435061] req@ffff9811c8cd6680 x1876350913888768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435082 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435061.036360:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913888768/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435082 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435061.036382:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435061.036387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435061.036391:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913888768:192.168.203.128@tcp:250:kworker.0 00000800:00000100:0.0:1789435061.038379:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789435061.038402:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:0.0:1789435061.038411:0:2377:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:1.0:1789435061.039798:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435057/real 1789435061] req@ffff9811c8cd5c00 x1876350913889024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435061.039829:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913889024/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435061.040390:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435061.040396:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435061.040401:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889024:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435061.040409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435057/real 1789435061] req@ffff9811c8cd7100 x1876350913889152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435061.040425:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913889152/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435061.040433:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435061.040436:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435061.040439:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889152:192.168.203.128@tcp:8:kworker.0 00000100:00100000:1.0:1789435061.040446:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435057/real 1789435061] req@ffff9811c8cd4700 x1876350913888896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435061.040457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913888896/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435061.040466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435061.040469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435061.040472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913888896:192.168.203.128@tcp:38:kworker.0 00010000:00010000:3.0:1789435061.926874:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435062.950116:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435063.081586:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435063.081595:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435063.081600:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435063.081619:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7432 00000100:00080000:1.0:1789435063.081629:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435063.081643:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435063.081953:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913889280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435063.081982:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435063.081986:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435063.081988:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435063.081992:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7432 00000100:00080000:1.0:1789435063.081996:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435063.082002:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435063.082011:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435063.082026:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913889408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435063.082036:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435063.082039:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435063.082041:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435063.082044:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7432 00000100:00100000:0.0:1789435063.082046:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913889280:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789435063.082048:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435063.082052:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435063.082056:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435063.082064:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913889536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435063.082074:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435063.082077:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435063.082078:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435063.082081:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7432 00000100:00080000:1.0:1789435063.082091:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435063.082095:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435063.082099:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000800:00000100:0.0:1789435063.082104:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435063.082108:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913889664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789435063.082139:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435063/real 1789435063] req@ffff9811d0bde300 x1876350913889280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435063.082166:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913889280/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435063.082177:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435063.082181:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435063.082185:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913889280:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435063.082192:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889408:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435063.082216:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435063.082228:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435063/real 1789435063] req@ffff9811d0bdf800 x1876350913889408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435063.082238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913889408/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435063.082246:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435063.082248:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435063.082251:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889408:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435063.082273:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889536:192.168.203.128@tcp:38:kworker.0 00000800:00000100:0.0:1789435063.082293:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435063.082303:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435063/real 1789435063] req@ffff9811d0bdf100 x1876350913889536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435063.082316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913889536/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435063.082321:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435063.082323:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435063.082325:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889536:192.168.203.128@tcp:38:kworker.0 00000100:00100000:0.0:1789435063.082328:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889664:192.168.203.128@tcp:8:kworker.0 00000800:00000100:0.0:1789435063.082346:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435063.082356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435063/real 1789435063] req@ffff9811d0bdea00 x1876350913889664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789435063.082367:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913889664/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789435063.082374:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789435063.082377:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:0.0:1789435063.082380:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889664:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435063.974804:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435064.998051:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435066.022754:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435067.046157:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435068.070563:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435068.198949:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435068.198959:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435068.198963:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435068.198973:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7437 00000100:00080000:0.0:1789435068.198982:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435068.198994:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435068.199030:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913889792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435068.199060:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435068.199063:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435068.199065:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435068.199069:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7437 00000100:00080000:0.0:1789435068.199073:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435068.199078:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435068.199088:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435068.199101:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913889920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435068.199111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435068.199114:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435068.199115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435068.199118:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7437 00000100:00080000:0.0:1789435068.199122:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435068.199126:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435068.199130:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435068.199140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913890048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435068.199149:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435068.199151:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435068.199153:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435068.199156:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7437 00000100:00080000:0.0:1789435068.199159:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435068.199163:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435068.199167:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435068.199177:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913890176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435068.199278:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913889792:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435068.199334:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435068.199382:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435068/real 1789435068] req@ffff9811c8cd6680 x1876350913889792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435068.199398:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913889792/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435068.199409:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435068.199413:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435068.199417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913889792:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435068.199424:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889920:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435068.199447:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435068.199459:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435068/real 1789435068] req@ffff9811c8cd5180 x1876350913889920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435068.199471:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913889920/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435068.199479:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435068.199482:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435068.199485:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913889920:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435068.199490:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890048:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435068.199510:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435068.199533:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435068/real 1789435068] req@ffff9811c8cd7800 x1876350913890048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435068.199544:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913890048/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435068.199551:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435068.199554:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435068.199557:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890048:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435068.199562:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890176:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435068.199581:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435068.199592:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435068/real 1789435068] req@ffff9811c8cd7480 x1876350913890176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435068.199603:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913890176/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435068.199611:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435068.199614:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435068.199617:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890176:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435069.095019:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435070.120665:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435071.142119:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435072.169016:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435073.190098:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435073.317935:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435073.317943:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435073.317948:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435073.317956:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7442 00000100:00080000:0.0:1789435073.317963:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435073.317973:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435073.318003:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913890304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435073.318029:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435073.318031:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435073.318033:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435073.318036:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7442 00000100:00080000:0.0:1789435073.318038:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435073.318040:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435073.318048:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435073.318056:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913890432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435073.318062:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435073.318064:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435073.318065:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435073.318067:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7442 00000100:00080000:0.0:1789435073.318069:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435073.318073:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435073.318076:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435073.318084:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913890560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435073.318099:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435073.318102:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435073.318104:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435073.318107:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7442 00000100:00080000:0.0:1789435073.318110:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435073.318113:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435073.318118:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435073.318128:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913890688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435073.318158:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913890304:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435073.318231:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435073.318285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435073/real 1789435073] req@ffff9811c8cd6a00 x1876350913890304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435073.318303:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913890304/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435073.318323:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435073.318328:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435073.318332:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913890304:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435073.318346:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890432:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435073.318369:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435073.318384:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435073/real 1789435073] req@ffff9811c8cd4000 x1876350913890432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435073.318402:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913890432/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435073.318411:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435073.318414:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435073.318417:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890432:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435073.318423:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890560:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435073.318441:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435073.318465:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435073/real 1789435073] req@ffff9811c8cd6300 x1876350913890560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435073.318492:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913890560/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435073.318502:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435073.318506:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435073.318509:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890560:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435073.318515:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890688:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435073.318535:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435073.318546:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435073/real 1789435073] req@ffff9811c8cd7b80 x1876350913890688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435073.318557:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913890688/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435073.318570:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435073.318573:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435073.318577:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890688:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435074.216986:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435075.238103:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435076.262887:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435077.286092:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435078.310014:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435078.437945:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435078.437955:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435078.437964:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435078.437977:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7448 00000100:00080000:0.0:1789435078.437986:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435078.437997:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435078.438029:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913890816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435078.438057:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435078.438059:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435078.438060:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435078.438063:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7448 00000100:00080000:0.0:1789435078.438066:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435078.438069:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435078.438077:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435078.438088:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913890944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435078.438095:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435078.438098:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435078.438099:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435078.438101:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7448 00000100:00080000:0.0:1789435078.438103:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435078.438108:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435078.438108:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913890816:192.168.203.128@tcp:250:kworker.0 00000008:00100000:0.0:1789435078.438112:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435078.438123:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913891072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435078.438130:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435078.438133:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435078.438134:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435078.438136:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7448 00000100:00080000:0.0:1789435078.438138:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435078.438141:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435078.438144:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435078.438153:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913891200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789435078.438172:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435078.438223:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435078/real 1789435078] req@ffff9811c8cd4a80 x1876350913890816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435078.438238:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913890816/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435078.438247:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435078.438250:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435078.438254:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913890816:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435078.438260:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890944:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435078.438281:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435078.438293:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435078/real 1789435078] req@ffff9811c8cd5500 x1876350913890944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435078.438304:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913890944/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435078.438311:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435078.438314:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435078.438316:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913890944:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435078.438338:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891072:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435078.438356:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435078.438366:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435078/real 1789435078] req@ffff9811c8cd6d80 x1876350913891072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435078.438377:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913891072/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435078.438384:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435078.438387:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435078.438390:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891072:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435078.438396:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891200:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435078.438415:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435078.438424:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435078/real 1789435078] req@ffff9811c8cd5f80 x1876350913891200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435078.438457:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913891200/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435078.438466:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435078.438469:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435078.438472:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891200:192.168.203.128@tcp:8:kworker.0 00010000:00010000:0.0:1789435079.334042:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435080.358406:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435081.382470:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435082.406026:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435083.430115:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435083.558145:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435083.558164:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435083.558169:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435083.558179:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7453 00000100:00080000:0.0:1789435083.558187:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435083.558199:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435083.558239:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913891328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435083.558274:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435083.558278:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435083.558280:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435083.558284:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7453 00000100:00080000:0.0:1789435083.558295:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435083.558300:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435083.558309:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435083.558321:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913891456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435083.558330:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435083.558333:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435083.558335:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435083.558339:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7453 00000100:00080000:0.0:1789435083.558342:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435083.558346:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435083.558351:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435083.558361:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913891584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435083.558371:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435083.558374:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435083.558375:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435083.558378:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7453 00000100:00080000:0.0:1789435083.558381:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435083.558533:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435083.558540:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435083.558562:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913891712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789435083.558670:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913891328:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789435083.558736:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435083.558787:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435083/real 1789435083] req@ffff9811c8cd5f80 x1876350913891328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435083.558805:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913891328/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435083.558826:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435083.558858:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435083.558863:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913891328:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789435083.558870:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891456:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435083.558894:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435083.558907:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435083/real 1789435083] req@ffff9811c8cd6d80 x1876350913891456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435083.558925:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913891456/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435083.558934:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435083.558938:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435083.558941:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891456:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435083.558948:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891584:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435083.558980:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435083.558991:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435083/real 1789435083] req@ffff9811c8cd5500 x1876350913891584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435083.559002:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913891584/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435083.559010:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435083.559013:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435083.559016:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891584:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435083.559022:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891712:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789435083.559041:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435083.559051:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435083/real 1789435083] req@ffff9811c8cd4a80 x1876350913891712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435083.559062:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913891712/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435083.559088:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435083.559091:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435083.559094:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891712:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435084.454069:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435085.478242:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435086.502866:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435087.529440:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435088.550262:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435088.679483:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435088.679493:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435088.679497:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435088.679507:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7458 00000100:00080000:1.0:1789435088.679516:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435088.679538:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435088.679573:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913891840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435088.679615:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435088.679619:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435088.679621:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435088.679625:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7458 00000100:00080000:1.0:1789435088.679629:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435088.679634:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435088.679643:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435088.679671:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913891968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435088.679680:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435088.679683:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435088.679685:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435088.679688:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7458 00000100:00080000:1.0:1789435088.679691:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435088.679696:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435088.679700:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435088.679710:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913892096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435088.679725:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435088.679728:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435088.679730:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435088.679734:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7458 00000100:00080000:1.0:1789435088.679737:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435088.679741:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435088.679745:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435088.679755:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913892224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789435088.679953:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913891840:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789435088.680011:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435088.680059:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435088/real 1789435088] req@ffff9811d0bdc700 x1876350913891840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435088.680076:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913891840/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435088.680085:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435088.680089:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435088.680092:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913891840:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789435088.680098:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891968:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435088.680120:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435088.680132:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435088/real 1789435088] req@ffff9811d0bdc000 x1876350913891968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435088.680143:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913891968/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435088.680150:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435088.680154:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435088.680157:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913891968:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435088.680162:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892096:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435088.680179:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435088.680189:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435088/real 1789435088] req@ffff9811d0bdd880 x1876350913892096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435088.680199:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913892096/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435088.680206:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435088.680209:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435088.680211:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892096:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435088.680216:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892224:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789435088.680233:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435088.680241:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435088/real 1789435088] req@ffff9811d0bde680 x1876350913892224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435088.680250:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913892224/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435088.680257:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435088.680259:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435088.680262:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892224:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435089.574142:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435090.599093:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789435091.622305:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435092.646112:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435093.670064:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435093.797977:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435093.797986:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435093.797990:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435093.798000:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7463 00000100:00080000:1.0:1789435093.798008:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435093.798018:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435093.798047:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913892352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435093.798070:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435093.798073:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435093.798075:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435093.798078:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7463 00000100:00080000:1.0:1789435093.798080:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435093.798082:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435093.798091:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435093.798098:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913892480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435093.798104:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435093.798106:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435093.798107:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435093.798109:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7463 00000100:00080000:1.0:1789435093.798110:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435093.798112:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435093.798114:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435093.798119:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913892608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435093.798125:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435093.798127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435093.798128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435093.798130:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7463 00000100:00080000:1.0:1789435093.798132:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435093.798134:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435093.798136:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435093.798140:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913892736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789435093.798196:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913892352:192.168.203.128@tcp:250:kworker.0 00000800:00000100:2.0:1789435093.798240:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435093.798276:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435093/real 1789435093] req@ffff9811d0bdf480 x1876350913892352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435093.798287:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913892352/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435093.798294:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435093.798296:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435093.798298:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913892352:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789435093.798302:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892480:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435093.798315:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435093.798321:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435093/real 1789435093] req@ffff9811d0bdfb80 x1876350913892480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435093.798341:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913892480/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435093.798349:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435093.798352:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435093.798355:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892480:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435093.798361:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892608:192.168.203.128@tcp:38:kworker.0 00000800:00000100:2.0:1789435093.798381:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435093.798391:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435093/real 1789435093] req@ffff9811d0bded80 x1876350913892608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435093.798401:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913892608/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435093.798406:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435093.798408:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435093.798410:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892608:192.168.203.128@tcp:38:kworker.0 00000100:00100000:2.0:1789435093.798413:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892736:192.168.203.128@tcp:8:kworker.0 00000800:00000100:2.0:1789435093.798425:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:2.0:1789435093.798431:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435093/real 1789435093] req@ffff9811d0bdc380 x1876350913892736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789435093.798437:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913892736/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435093.798446:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789435093.798448:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435093.798450:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892736:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435094.694992:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435095.718052:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435096.742778:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435097.766334:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435098.790896:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435098.918943:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435098.918954:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435098.918958:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435098.918969:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7468 00000100:00080000:2.0:1789435098.918978:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435098.918989:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435098.919026:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913892864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435098.919091:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913892864:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435098.919153:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789435098.919184:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435098.919188:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435098.919190:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435098.919195:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7468 00000100:00080000:2.0:1789435098.919199:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435098.919205:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435098.919215:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435098.919229:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913892992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435098.919299:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435098/real 1789435098] req@ffff9811d2101180 x1876350913892864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435098.919318:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913892864/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435098.919328:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435098.919333:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435098.919336:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913892864:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435098.919360:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892992:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435098.919384:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435098.919397:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435098/real 1789435098] req@ffff9811d2101f80 x1876350913892992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435098.919409:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913892992/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435098.919416:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435098.919419:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435098.919422:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913892992:192.168.203.128@tcp:38:kworker.0 00000100:00080000:2.0:1789435098.919518:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435098.919523:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435098.919525:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435098.919529:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7468 00000100:00080000:2.0:1789435098.919533:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435098.919539:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435098.919544:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435098.919557:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913893120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435098.919602:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893120:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435098.919625:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789435098.919649:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435098.919652:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435098.919654:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435098.919659:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7468 00000100:00080000:2.0:1789435098.919663:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435098.919668:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435098.919673:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789435098.919687:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913893248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435098.919726:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435098/real 1789435098] req@ffff9811d2100e00 x1876350913893120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435098.919738:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913893120/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435098.919751:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435098.919754:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435098.919758:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893120:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435098.919770:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893248:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435098.919791:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435098.919801:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435098/real 1789435098] req@ffff9811d2101f80 x1876350913893248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435098.919811:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913893248/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435098.919819:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435098.919822:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435098.919824:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893248:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435099.818598:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435100.838073:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435101.862193:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435102.886053:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435103.910585:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435104.037923:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435104.037932:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435104.037937:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435104.037947:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7473 00000100:00080000:2.0:1789435104.037956:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435104.037967:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435104.038000:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913893376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435104.038024:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435104.038027:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435104.038039:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435104.038044:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7473 00000100:00080000:2.0:1789435104.038048:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789435104.038053:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913893376:192.168.203.128@tcp:250:kworker.0 00000100:00080000:2.0:1789435104.038055:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435104.038068:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435104.038083:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913893504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435104.038104:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435104.038108:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435104.038110:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435104.038114:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7473 00000800:00000100:1.0:1789435104.038119:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:2.0:1789435104.038119:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435104.038126:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435104.038131:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435104.038154:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913893632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435104.038164:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435104.038168:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1789435104.038172:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435104/real 1789435104] req@ffff9811d2101180 x1876350913893376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789435104.038172:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435104.038180:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7473 00000100:00080000:2.0:1789435104.038187:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435104.038193:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435104.038198:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913893376/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789435104.038199:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00080000:1.0:1789435104.038211:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435104.038214:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:2.0:1789435104.038219:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913893760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435104.038220:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913893376:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435104.038245:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893504:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435104.038274:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435104.038285:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435104/real 1789435104] req@ffff9811d2102a00 x1876350913893504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435104.038293:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913893504/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435104.038299:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435104.038301:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435104.038303:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893504:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435104.038307:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893632:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435104.038317:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435104.038322:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435104/real 1789435104] req@ffff9811d2101500 x1876350913893632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435104.038329:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913893632/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435104.038334:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435104.038335:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435104.038337:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893632:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435104.038340:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893760:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435104.038351:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435104.038356:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435104/real 1789435104] req@ffff9811d2101880 x1876350913893760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435104.038363:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913893760/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435104.038367:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435104.038369:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435104.038370:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913893760:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435104.934582:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435105.958319:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435106.981976:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435108.006132:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435109.030039:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435109.157994:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435109.158008:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435109.158013:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435109.158024:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7478 00000100:00080000:2.0:1789435109.158032:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435109.158043:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435109.158077:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913893888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435109.158107:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435109.158111:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435109.158112:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435109.158116:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7478 00000100:00080000:2.0:1789435109.158120:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435109.158124:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435109.158134:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435109.158145:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913894016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435109.158154:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435109.158157:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435109.158158:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435109.158162:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7478 00000100:00080000:2.0:1789435109.158165:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435109.158169:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435109.158172:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435109.158181:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913894144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435109.158197:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435109.158200:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435109.158202:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435109.158206:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7478 00000100:00080000:2.0:1789435109.158209:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435109.158228:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435109.158234:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435109.158235:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913893888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:2.0:1789435109.158247:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913894272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789435109.158297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435109.158332:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435109/real 1789435109] req@ffff9811d2103b80 x1876350913893888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435109.158349:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913893888/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435109.158359:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435109.158374:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435109.158377:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913893888:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435109.158384:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894016:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435109.158404:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435109.158426:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435109/real 1789435109] req@ffff9811d2101c00 x1876350913894016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435109.158438:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913894016/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435109.158451:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435109.158454:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435109.158458:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894016:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435109.158472:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894144:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435109.158669:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435109.158682:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435109/real 1789435109] req@ffff9811d2100000 x1876350913894144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435109.158741:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913894144/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435109.158748:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435109.158752:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435109.158755:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894144:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435109.158778:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894272:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435109.158795:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435109.158803:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435109/real 1789435109] req@ffff9811d2102d80 x1876350913894272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435109.158810:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913894272/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435109.158815:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435109.158817:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435109.158819:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894272:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435110.055021:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435111.078081:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435112.102188:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435113.126102:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435114.150312:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435114.278319:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435114.278330:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435114.278334:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435114.278344:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7483 00000100:00080000:1.0:1789435114.278355:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435114.278366:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435114.278403:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd500 x1876350913894400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435114.278435:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435114.278438:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435114.278440:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435114.278445:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7483 00000100:00080000:1.0:1789435114.278449:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435114.278453:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435114.278463:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435114.278474:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bddc00 x1876350913894528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435114.278483:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435114.278486:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435114.278487:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435114.278490:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7483 00000100:00080000:1.0:1789435114.278493:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435114.278499:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435114.278503:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435114.278511:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd180 x1876350913894656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435114.278520:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435114.278523:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435114.278524:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435114.278527:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7483 00000100:00080000:1.0:1789435114.278530:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435114.278534:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435114.278538:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435114.278550:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc380 x1876350913894784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435114.278913:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913894400:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435114.278970:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435114.279007:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435114/real 1789435114] req@ffff9811d0bdd500 x1876350913894400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435114.279026:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd500 x1876350913894400/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435114.279036:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435114.279040:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435114.279044:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913894400:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435114.279052:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894528:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435114.279074:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435114.279086:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435114/real 1789435114] req@ffff9811d0bddc00 x1876350913894528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435114.279099:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bddc00 x1876350913894528/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435114.279107:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435114.279111:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435114.279114:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bddc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894528:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435114.279120:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894656:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435114.279139:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435114.279151:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435114/real 1789435114] req@ffff9811d0bdd180 x1876350913894656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435114.279168:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd180 x1876350913894656/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435114.279189:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435114.279193:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435114.279196:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894656:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435114.279202:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894784:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435114.279222:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435114.279233:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435114/real 1789435114] req@ffff9811d0bdc380 x1876350913894784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435114.279245:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc380 x1876350913894784/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435114.279253:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435114.279256:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435114.279259:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913894784:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435115.174271:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435116.198299:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435117.222082:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435118.246199:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435119.270044:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435119.398189:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435119.398201:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435119.398205:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435119.398214:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7488 00000100:00080000:0.0:1789435119.398223:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435119.398235:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435119.398275:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913894912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435119.398306:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435119.398310:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435119.398312:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435119.398316:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7488 00000100:00080000:0.0:1789435119.398319:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435119.398324:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435119.398334:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435119.398346:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913895040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435119.398354:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435119.398358:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435119.398359:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435119.398362:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7488 00000100:00080000:0.0:1789435119.398364:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435119.398368:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435119.398372:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435119.398381:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913895168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435119.398390:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435119.398394:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435119.398396:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435119.398400:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7488 00000100:00080000:0.0:1789435119.398403:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789435119.398408:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913894912:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789435119.398412:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435119.398420:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435119.398431:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913895296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789435119.398488:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435119.398531:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435119/real 1789435119] req@ffff9811c8cd7b80 x1876350913894912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435119.398558:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913894912/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435119.398567:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435119.398571:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435119.398575:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913894912:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435119.398583:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895040:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435119.398612:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435119.398624:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435119/real 1789435119] req@ffff9811c8cd6300 x1876350913895040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435119.398636:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913895040/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435119.398654:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435119.398658:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435119.398661:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895040:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435119.398666:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895168:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435119.398695:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435119.398705:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435119/real 1789435119] req@ffff9811c8cd4000 x1876350913895168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435119.398715:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913895168/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435119.398727:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435119.398730:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435119.398733:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895168:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435119.398760:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895296:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435119.398780:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435119.398790:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435119/real 1789435119] req@ffff9811c8cd6a00 x1876350913895296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435119.398812:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913895296/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435119.398821:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435119.398824:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435119.398827:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895296:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435120.296222:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435121.317991:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435122.342009:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435123.367829:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435124.389941:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435124.518023:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435124.518038:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435124.518044:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435124.518056:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7494 00000100:00080000:1.0:1789435124.518071:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435124.518086:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435124.518153:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bded80 x1876350913895424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435124.518248:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913895424:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789435124.518385:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435124.518390:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435124.518393:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435124.518399:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7494 00000100:00080000:1.0:1789435124.518404:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435124.518410:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435124.518422:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435124.518440:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdfb80 x1876350913895552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435124.518525:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895552:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789435124.518610:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435124.518618:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435124.518620:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435124.518624:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7494 00000100:00080000:1.0:1789435124.518629:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435124.518635:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435124.518641:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435124.518670:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf480 x1876350913895680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435124.518691:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435124.518695:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435124.518696:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435124.518700:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7494 00000100:00080000:1.0:1789435124.518704:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435124.518709:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435124.518715:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:3.0:1789435124.518727:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435124.518731:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde680 x1876350913895808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435124.518770:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895808:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435125.414493:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435126.438380:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435127.462210:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789435127.590524:0:2378:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.128:988 00000400:00000100:1.0:1789435127.590579:0:2378:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.128@tcp at host 192.168.203.128 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789435127.590598:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789435127.590652:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789435127.590661:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000800:00000100:1.0:1789435127.590669:0:2378:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.128@tcp 00000100:00100000:3.0:1789435127.590944:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435124/real 1789435127] req@ffff9811d0bde680 x1876350913895808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435127.590972:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde680 x1876350913895808/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435127.590984:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435127.590989:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435127.590993:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895808:192.168.203.128@tcp:8:kworker.0 00000100:00100000:3.0:1789435127.591001:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435124/real 1789435127] req@ffff9811d0bdf480 x1876350913895680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435127.591011:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf480 x1876350913895680/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435127.591018:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435127.591030:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435127.591034:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895680:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435127.591039:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435124/real 1789435127] req@ffff9811d0bdfb80 x1876350913895552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435127.591049:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdfb80 x1876350913895552/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435127.591056:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435127.591059:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435127.591062:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913895552:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435127.591069:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435124/real 1789435127] req@ffff9811d0bded80 x1876350913895424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435127.591088:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bded80 x1876350913895424/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435127.591095:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435127.591098:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435127.591101:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bded80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913895424:192.168.203.128@tcp:250:kworker.0 00010000:00010000:3.0:1789435128.486275:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435129.510010:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435129.640664:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435129.640676:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435129.640680:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435129.640699:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7499 00000100:00080000:2.0:1789435129.640709:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435129.640719:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435129.640752:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103800 x1876350913895936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435129.640783:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435129.640787:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435129.640789:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435129.640793:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7499 00000100:00080000:2.0:1789435129.640797:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435129.640806:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435129.640817:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435129.640857:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102d80 x1876350913896064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435129.640867:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435129.640870:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435129.640872:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435129.640875:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7499 00000100:00080000:2.0:1789435129.640878:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435129.640882:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435129.640886:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435129.640894:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100000 x1876350913896192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435129.640902:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435129.640905:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435129.640906:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435129.640910:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7499 00000100:00080000:2.0:1789435129.640912:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435129.640916:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435129.640921:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789435129.640930:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101c00 x1876350913896320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435129.641114:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913895936:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435129.641178:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435129.641215:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435129/real 1789435129] req@ffff9811d2103800 x1876350913895936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435129.641233:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103800 x1876350913895936/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435129.641242:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435129.641246:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435129.641250:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913895936:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435129.641257:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896064:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435129.641276:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435129.641286:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435129/real 1789435129] req@ffff9811d2102d80 x1876350913896064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435129.641304:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102d80 x1876350913896064/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435129.641313:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435129.641315:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435129.641318:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896064:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435129.641323:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896192:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435129.641338:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435129.641347:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435129/real 1789435129] req@ffff9811d2100000 x1876350913896192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435129.641357:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100000 x1876350913896192/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435129.641363:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435129.641381:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435129.641384:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896192:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435129.641390:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896320:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435129.641407:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435129.641417:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435129/real 1789435129] req@ffff9811d2101c00 x1876350913896320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435129.641426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101c00 x1876350913896320/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435129.641438:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435129.641441:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435129.641444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896320:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435130.534107:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435131.558034:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435132.587766:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435133.606932:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435134.630069:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435134.761535:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435134.761546:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435134.761550:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435134.761561:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7504 00000100:00080000:0.0:1789435134.761570:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435134.761582:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435134.761620:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913896448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435134.761651:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435134.761655:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435134.761657:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435134.761661:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7504 00000100:00080000:0.0:1789435134.761665:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435134.761671:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435134.761681:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435134.761697:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913896576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435134.761701:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913896448:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789435134.761706:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435134.761709:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435134.761711:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435134.761714:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7504 00000100:00080000:0.0:1789435134.761717:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435134.761721:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435134.761725:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435134.761734:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913896704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435134.761742:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435134.761745:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435134.761746:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435134.761749:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7504 00000100:00080000:0.0:1789435134.761752:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435134.761756:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:3.0:1789435134.761756:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000008:00100000:0.0:1789435134.761760:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435134.761769:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913896832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435134.761791:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435134/real 1789435134] req@ffff9811c8cd7480 x1876350913896448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435134.761809:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913896448/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435134.761819:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435134.761822:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435134.761826:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913896448:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435134.761871:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896576:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435134.761893:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435134.761905:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435134/real 1789435134] req@ffff9811c8cd7800 x1876350913896576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435134.761917:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913896576/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435134.761930:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435134.761933:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435134.761936:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896576:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435134.761942:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896704:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435134.761961:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435134.761972:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435134/real 1789435134] req@ffff9811c8cd5180 x1876350913896704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435134.761983:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913896704/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435134.761990:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435134.761993:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435134.761996:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896704:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435134.762002:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896832:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435134.762020:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435134.762042:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435134/real 1789435134] req@ffff9811c8cd6680 x1876350913896832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435134.762053:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913896832/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435134.762061:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435134.762063:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435134.762067:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913896832:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435135.654085:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435136.678178:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435137.702030:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435138.726397:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435139.750067:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435139.877946:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435139.877956:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435139.877959:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435139.877970:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7509 00000100:00080000:0.0:1789435139.877978:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435139.878003:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435139.878037:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4700 x1876350913896960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435139.878075:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435139.878081:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435139.878083:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435139.878088:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7509 00000100:00080000:0.0:1789435139.878091:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435139.878095:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435139.878105:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435139.878116:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7100 x1876350913897088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435139.878124:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435139.878127:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435139.878128:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435139.878131:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7509 00000100:00080000:0.0:1789435139.878134:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435139.878138:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435139.878141:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435139.878149:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5c00 x1876350913897216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435139.878158:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435139.878161:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435139.878163:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435139.878165:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7509 00000100:00080000:0.0:1789435139.878168:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435139.878172:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435139.878177:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435139.878187:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6680 x1876350913897344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435139.878280:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913896960:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435139.878335:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435139.878370:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435139/real 1789435139] req@ffff9811c8cd4700 x1876350913896960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435139.878386:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4700 x1876350913896960/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435139.878396:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435139.878399:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435139.878403:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913896960:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435139.878409:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897088:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435139.878429:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435139.878453:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435139/real 1789435139] req@ffff9811c8cd7100 x1876350913897088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435139.878464:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7100 x1876350913897088/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435139.878471:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435139.878474:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435139.878477:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897088:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435139.878494:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897216:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435139.878513:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435139.878524:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435139/real 1789435139] req@ffff9811c8cd5c00 x1876350913897216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435139.878535:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5c00 x1876350913897216/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435139.878543:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435139.878546:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435139.878549:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897216:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435139.878554:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897344:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435139.878572:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435139.878583:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435139/real 1789435139] req@ffff9811c8cd6680 x1876350913897344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435139.878594:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6680 x1876350913897344/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435139.878602:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435139.878605:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435139.878608:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897344:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435140.774186:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435141.798009:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435142.822346:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435143.847332:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435144.870829:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435144.998020:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435144.998029:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435144.998032:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435144.998041:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7514 00000100:00080000:0.0:1789435144.998049:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435144.998058:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435144.998089:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5180 x1876350913897472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435144.999319:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913897472:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789435144.999340:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435144.999343:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435144.999346:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435144.999350:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7514 00000100:00080000:0.0:1789435144.999354:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435144.999359:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435144.999368:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789435144.999375:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:0.0:1789435144.999382:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913897600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435144.999409:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435144/real 1789435144] req@ffff9811c8cd5180 x1876350913897472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435144.999426:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5180 x1876350913897472/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435144.999436:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435144.999440:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435144.999444:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913897472:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435144.999464:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897600:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435144.999485:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435144.999497:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435144/real 1789435144] req@ffff9811c8cd7800 x1876350913897600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435144.999509:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913897600/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435144.999518:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435144.999520:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435144.999523:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897600:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789435144.999549:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435144.999554:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435144.999556:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435144.999560:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7514 00000100:00080000:0.0:1789435144.999564:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435144.999570:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435144.999575:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435144.999586:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913897728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435144.999608:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789435144.999783:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897728:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435144.999803:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435144.999814:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435144/real 1789435144] req@ffff9811c8cd7480 x1876350913897728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435144.999856:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913897728/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435144.999864:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435144.999867:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435144.999870:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897728:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789435144.999927:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435144.999928:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435144.999933:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7514 00000100:00080000:0.0:1789435144.999937:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435144.999941:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435144.999959:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435144.999972:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913897856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435145.000047:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897856:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435145.000068:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435145.000079:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435144/real 1789435145] req@ffff9811c8cd6a00 x1876350913897856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435145.000089:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913897856/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435145.000097:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435145.000100:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435145.000103:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913897856:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435145.894089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435146.918077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435147.943923:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435148.967015:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435149.991739:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435150.125940:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435150.125952:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435150.125965:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435150.125975:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7519 00000100:00080000:0.0:1789435150.125985:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435150.125997:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435150.126030:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913897984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435150.126060:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435150.126065:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435150.126066:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435150.126078:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7519 00000100:00080000:0.0:1789435150.126081:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435150.126085:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435150.126094:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435150.126104:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913898112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435150.126113:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435150.126115:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435150.126117:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435150.126119:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7519 00000100:00080000:0.0:1789435150.126121:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435150.126124:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435150.126127:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435150.126133:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913898240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435150.126140:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435150.126146:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435150.126147:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435150.126158:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7519 00000100:00080000:0.0:1789435150.126161:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435150.126165:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435150.126170:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435150.127368:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913898368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435150.127686:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913897984:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435150.127746:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435150.127803:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435150/real 1789435150] req@ffff9811c8cd4000 x1876350913897984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435150.127827:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913897984/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435150.127868:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435150.127873:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435150.127876:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913897984:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435150.127885:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898112:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435150.127919:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435150.127932:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435150/real 1789435150] req@ffff9811c8cd6300 x1876350913898112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435150.127943:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913898112/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435150.127951:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435150.127954:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435150.127956:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898112:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435150.127962:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898240:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435150.127985:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435150.127996:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435150/real 1789435150] req@ffff9811c8cd7b80 x1876350913898240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435150.128006:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913898240/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435150.128017:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435150.128020:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435150.128023:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898240:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435150.128028:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898368:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435150.128046:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435150.128056:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435150/real 1789435150] req@ffff9811c8cd4a80 x1876350913898368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435150.128067:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913898368/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435150.128086:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435150.128089:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435150.128092:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898368:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435151.014711:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435152.038674:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435153.066967:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435154.089123:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435155.111080:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435155.244063:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435155.244081:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435155.244085:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435155.244095:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7524 00000100:00080000:2.0:1789435155.244113:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435155.244123:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435155.244159:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2103b80 x1876350913898496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435155.244190:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435155.244194:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435155.244196:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435155.244200:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7524 00000100:00080000:2.0:1789435155.244203:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435155.244207:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435155.244217:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435155.244228:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101880 x1876350913898624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435155.244237:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435155.244240:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435155.244241:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435155.244244:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7524 00000100:00080000:2.0:1789435155.244246:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435155.244249:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435155.244253:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435155.244260:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101500 x1876350913898752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435155.244268:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435155.244271:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435155.244272:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435155.244274:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7524 00000100:00080000:2.0:1789435155.244276:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435155.244280:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435155.244284:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789435155.244291:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913898880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435155.244599:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913898496:192.168.203.128@tcp:250:kworker.0 00000800:00000100:1.0:1789435155.244665:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435155.244704:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435155/real 1789435155] req@ffff9811d2103b80 x1876350913898496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435155.244718:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2103b80 x1876350913898496/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435155.244727:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435155.244731:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435155.244734:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2103b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913898496:192.168.203.128@tcp:250:kworker.0 00000100:00100000:1.0:1789435155.244740:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898624:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435155.244760:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435155.244771:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435155/real 1789435155] req@ffff9811d2101880 x1876350913898624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435155.244781:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101880 x1876350913898624/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435155.244792:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435155.244795:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435155.244797:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898624:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435155.244802:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898752:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435155.244818:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435155.244827:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435155/real 1789435155] req@ffff9811d2101500 x1876350913898752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435155.244878:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101500 x1876350913898752/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435155.244887:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435155.244890:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435155.244893:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898752:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435155.244899:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898880:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435155.244918:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435155.244928:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435155/real 1789435155] req@ffff9811d2102a00 x1876350913898880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435155.244938:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913898880/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435155.244945:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435155.244947:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435155.244950:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913898880:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435156.134164:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435157.158290:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435158.182398:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435159.206075:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435160.233929:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435160.358335:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435160.358346:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435160.358407:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435160.358418:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7529 00000100:00080000:0.0:1789435160.358427:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435160.358439:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435160.358474:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913899008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435160.358499:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435160.358503:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435160.358504:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435160.358508:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7529 00000100:00080000:0.0:1789435160.358511:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435160.358516:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435160.358525:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435160.358540:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6d80 x1876350913899136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435160.358541:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913899008:192.168.203.128@tcp:250:kworker.0 00000100:00080000:0.0:1789435160.358555:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435160.358558:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435160.358561:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435160.358564:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7529 00000100:00080000:0.0:1789435160.358568:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435160.358584:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000800:00000100:1.0:1789435160.358593:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435160.358630:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435160/real 1789435160] req@ffff9811c8cd5500 x1876350913899008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435160.358642:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913899008/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789435160.358651:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789435160.358651:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435160.358655:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435160.358659:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913899008:192.168.203.128@tcp:250:kworker.0 00000100:00100000:0.0:1789435160.358668:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5f80 x1876350913899264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435160.358681:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435160.358685:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435160.358686:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435160.358690:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7529 00000100:00080000:0.0:1789435160.358694:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435160.358699:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435160.358703:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435160.358714:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd5500 x1876350913899392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789435160.358772:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899136:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435160.358791:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435160.358800:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435160/real 1789435160] req@ffff9811c8cd6d80 x1876350913899136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435160.358808:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6d80 x1876350913899136/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435160.358813:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435160.358815:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435160.358818:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899136:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435160.358826:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899264:192.168.203.128@tcp:38:kworker.0 00000800:00000100:1.0:1789435160.358875:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435160.358883:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435160/real 1789435160] req@ffff9811c8cd5f80 x1876350913899264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435160.358890:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5f80 x1876350913899264/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435160.358895:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435160.358897:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435160.358900:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899264:192.168.203.128@tcp:38:kworker.0 00000100:00100000:1.0:1789435160.358903:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899392:192.168.203.128@tcp:8:kworker.0 00000800:00000100:1.0:1789435160.358914:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435160.358919:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435160/real 1789435160] req@ffff9811c8cd5500 x1876350913899392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435160.358925:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd5500 x1876350913899392/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435160.358930:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789435160.358932:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:1.0:1789435160.358934:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899392:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435161.254050:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435162.280873:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435163.302220:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435164.326972:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435165.350475:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435165.479283:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435165.479293:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435165.479297:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435165.479308:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7535 00000100:00080000:1.0:1789435165.479317:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435165.479329:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435165.479880:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdd880 x1876350913899520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435165.479909:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435165.479914:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435165.479916:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435165.479920:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7535 00000100:00080000:1.0:1789435165.479925:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435165.479931:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435165.479942:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435165.479957:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913899520:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435165.480016:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:1.0:1789435165.480027:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc000 x1876350913899648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435165.480041:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435165.480044:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435165.480046:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435165.480052:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7535 00000100:00100000:3.0:1789435165.480053:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435165/real 1789435165] req@ffff9811d0bdd880 x1876350913899520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789435165.480058:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435165.480067:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435165.480073:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789435165.480078:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdd880 x1876350913899520/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789435165.480088:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913899776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789435165.480089:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435165.480097:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435165.480104:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913899520:192.168.203.128@tcp:250:kworker.0 00000100:00080000:1.0:1789435165.480108:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435165.480113:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435165.480115:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435165.480120:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7535 00000100:00080000:1.0:1789435165.480124:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789435165.480128:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899648:192.168.203.128@tcp:38:kworker.0 00000100:00080000:1.0:1789435165.480132:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435165.480138:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435165.480151:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913899904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789435165.480155:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435165.480169:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435165/real 1789435165] req@ffff9811d0bdc000 x1876350913899648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435165.480182:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc000 x1876350913899648/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435165.480190:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435165.480194:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435165.480198:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899648:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435165.480203:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899776:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435165.480230:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435165.480239:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435165/real 1789435165] req@ffff9811d0bdc700 x1876350913899776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435165.480257:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913899776/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435165.480264:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435165.480267:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435165.480270:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899776:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435165.480281:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899904:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435165.480297:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435165.480306:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435165/real 1789435165] req@ffff9811d0bdea00 x1876350913899904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435165.480316:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913899904/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435165.480325:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435165.480327:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435165.480330:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913899904:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435166.374049:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435167.398046:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435168.422536:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435169.445934:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435170.470442:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435170.597979:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435170.597989:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435170.597994:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435170.598013:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7540 00000100:00080000:1.0:1789435170.598022:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435170.598033:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:1.0:1789435170.598067:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf100 x1876350913900032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435170.598111:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435170.598115:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789435170.598116:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435170.598128:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7540 00000100:00080000:1.0:1789435170.598132:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435170.598140:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435170.598149:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435170.598162:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdf800 x1876350913900160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435170.598171:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435170.598175:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1789435170.598176:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435170.598179:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7540 00000100:00080000:1.0:1789435170.598182:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435170.598186:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435170.598190:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789435170.598199:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bde300 x1876350913900288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789435170.598209:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435170.598212:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1789435170.598213:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435170.598217:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7540 00000100:00080000:1.0:1789435170.598220:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435170.598223:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:1.0:1789435170.598227:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:1.0:1789435170.598239:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdea00 x1876350913900416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435170.598644:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913900032:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435170.598698:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435170.598733:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435170/real 1789435170] req@ffff9811d0bdf100 x1876350913900032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435170.598754:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf100 x1876350913900032/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435170.598775:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435170.598779:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435170.598783:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913900032:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435170.598790:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900160:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435170.598812:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435170.598824:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435170/real 1789435170] req@ffff9811d0bdf800 x1876350913900160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435170.598861:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdf800 x1876350913900160/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435170.598868:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435170.598872:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435170.598875:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900160:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435170.598880:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900288:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435170.598899:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435170.598909:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435170/real 1789435170] req@ffff9811d0bde300 x1876350913900288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435170.598920:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bde300 x1876350913900288/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435170.598933:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435170.598937:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435170.598940:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bde300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900288:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435170.598946:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900416:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435170.598964:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435170.598973:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435170/real 1789435170] req@ffff9811d0bdea00 x1876350913900416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435170.598984:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdea00 x1876350913900416/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435170.598992:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435170.598995:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435170.598998:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900416:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435171.494202:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435172.518004:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435173.542085:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435174.570561:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435175.590446:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789435175.717992:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435175.718002:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789435175.718006:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435175.718015:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7545 00000100:00080000:0.0:1789435175.718024:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435175.718036:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435175.718070:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4a80 x1876350913900544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435175.718102:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435175.718108:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435175.718110:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435175.718115:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7545 00000100:00080000:0.0:1789435175.718118:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435175.718123:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435175.718132:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435175.718146:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7b80 x1876350913900672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435175.718155:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435175.718158:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435175.718159:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435175.718163:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7545 00000100:00080000:0.0:1789435175.718166:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435175.718170:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435175.718174:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435175.718183:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6300 x1876350913900800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789435175.718193:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435175.718196:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435175.718198:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435175.718202:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7545 00000100:00080000:0.0:1789435175.718205:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435175.718209:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435175.718214:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435175.718224:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd4000 x1876350913900928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435175.718297:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913900544:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435175.718351:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435175.718386:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435175/real 1789435175] req@ffff9811c8cd4a80 x1876350913900544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435175.718403:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4a80 x1876350913900544/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435175.718423:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435175.718427:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435175.718430:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913900544:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435175.718437:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900672:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435175.718459:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435175.718471:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435175/real 1789435175] req@ffff9811c8cd7b80 x1876350913900672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435175.718483:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7b80 x1876350913900672/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435175.718491:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435175.718494:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435175.718497:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900672:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435175.718502:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900800:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435175.718518:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435175.718527:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435175/real 1789435175] req@ffff9811c8cd6300 x1876350913900800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435175.718537:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6300 x1876350913900800/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435175.718544:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435175.718547:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435175.718550:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900800:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435175.718556:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900928:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435175.718573:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435175.718582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435175/real 1789435175] req@ffff9811c8cd4000 x1876350913900928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435175.718593:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd4000 x1876350913900928/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435175.718600:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435175.718603:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435175.718606:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913900928:192.168.203.128@tcp:8:kworker.0 00010000:00010000:2.0:1789435176.614045:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435177.639077:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435178.662164:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435179.686452:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435180.710220:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789435180.845775:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789435180.845784:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789435180.845789:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789435180.845799:0:298853:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7550 00000100:00080000:1.0:1789435180.845808:0:298853:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789435180.845819:0:298853:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:0.0:1789435180.847229:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d0bdc700 x1876350913901056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435180.847379:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913901056:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435180.847424:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789435180.847449:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435180.847452:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789435180.847453:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435180.847457:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7550 00000100:00080000:0.0:1789435180.847460:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435180.847465:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435180.847473:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435180.847484:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd6a00 x1876350913901184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435180.847524:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435180/real 1789435180] req@ffff9811d0bdc700 x1876350913901056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435180.847535:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d0bdc700 x1876350913901056/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435180.847542:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435180.847544:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435180.847547:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d0bdc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913901056:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435180.847562:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901184:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435180.847576:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435180.847582:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435180/real 1789435180] req@ffff9811c8cd6a00 x1876350913901184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435180.847589:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd6a00 x1876350913901184/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435180.847594:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435180.847596:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435180.847598:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901184:192.168.203.128@tcp:38:kworker.0 00000100:00080000:0.0:1789435180.847771:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435180.847776:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1789435180.847778:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435180.847782:0:298853:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7550 00000100:00080000:0.0:1789435180.847786:0:298853:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435180.847791:0:298853:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435180.847796:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789435180.847809:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7480 x1876350913901312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435180.847861:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901312:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435180.847883:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00080000:0.0:1789435180.847921:0:298853:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789435180.847925:0:298853:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1789435180.847927:0:298853:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789435180.847931:0:298853:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7550 00000100:00080000:0.0:1789435180.847934:0:298853:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789435180.847939:0:298853:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:0.0:1789435180.847944:0:298853:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:0.0:1789435180.847955:0:298853:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811c8cd7800 x1876350913901440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435180.847991:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435180/real 1789435180] req@ffff9811c8cd7480 x1876350913901312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435180.848003:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7480 x1876350913901312/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435180.848010:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435180.848014:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435180.848017:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901312:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435180.848028:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901440:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435180.848047:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435180.848056:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435180/real 1789435180] req@ffff9811c8cd7800 x1876350913901440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435180.848072:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811c8cd7800 x1876350913901440/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435180.848080:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435180.848083:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435180.848086:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811c8cd7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901440:192.168.203.128@tcp:8:kworker.0 00010000:00010000:3.0:1789435181.746992:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435182.759089:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435183.782711:0:6631:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789435184.806102:0:6160:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789435185.830063:0:6161:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789435185.972389:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) 7564615c-5d86-4661-8fe3-c97a2f99cc3f->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435185.972399:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789435185.972403:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435185.972413:0:329502:0:(import.c:545:import_select_connection()) MGC192.168.203.128@tcp: connect to NID 192.168.203.128@tcp last attempt 7555 00000100:00080000:2.0:1789435185.972423:0:329502:0:(import.c:630:import_select_connection()) MGC192.168.203.128@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435185.972433:0:329502:0:(import.c:666:import_select_connection()) MGC192.168.203.128@tcp: import ffff9811d0dff800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000100:00100000:2.0:1789435185.972464:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101180 x1876350913901568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435185.972494:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435185.972497:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789435185.972499:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435185.972503:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7555 00000100:00080000:2.0:1789435185.972507:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435185.972511:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0000-mdc-ffff9811d1e45800: import ffff9811d20d1800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435185.972520:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435185.972531:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2101f80 x1876350913901696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435185.972541:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435185.972544:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1789435185.972546:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435185.972549:0:329502:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7555 00000100:00080000:2.0:1789435185.972552:0:329502:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435185.972555:0:329502:0:(import.c:666:import_select_connection()) lustre-MDT0001-mdc-ffff9811d1e45800: import ffff9811d20d2000 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435185.972559:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683433 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789435185.972567:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2100e00 x1876350913901824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789435185.972576:0:329502:0:(pinger.c:229:ptlrpc_pinger_process_import()) b53d9074-0ce4-44cb-9ce2-c2ae984122e2->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789435185.972579:0:329502:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1789435185.972580:0:329502:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789435185.972584:0:329502:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: connect to NID 192.168.203.128@tcp last attempt 7555 00000100:00080000:2.0:1789435185.972586:0:329502:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789435185.972589:0:329502:0:(import.c:666:import_select_connection()) lustre-OST0001-osc-ffff9811d1e45800: import ffff9811d1a44800 using connection 192.168.203.128@tcp/192.168.203.128@tcp 00000008:00100000:2.0:1789435185.972593:0:329502:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683433 ocd_grant: 28672, lost: 0. 00000100:00100000:2.0:1789435185.972602:0:329502:0:(import.c:871:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9811d2102a00 x1876350913901952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789435185.972775:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913901568:192.168.203.128@tcp:250:kworker.0 00000800:00000100:3.0:1789435185.972827:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435185.972885:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435185/real 1789435185] req@ffff9811d2101180 x1876350913901568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435185.972902:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101180 x1876350913901568/t0(0) o250->MGC192.168.203.128@tcp@192.168.203.128@tcp:26/25 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435185.972913:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d0dff800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435185.972916:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435185.972920:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:7564615c-5d86-4661-8fe3-c97a2f99cc3f:2388:1876350913901568:192.168.203.128@tcp:250:kworker.0 00000100:00100000:3.0:1789435185.972949:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901696:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435185.972968:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435185.972979:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435185/real 1789435185] req@ffff9811d2101f80 x1876350913901696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435185.972995:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2101f80 x1876350913901696/t0(0) o38->lustre-MDT0000-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435185.973003:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d1800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435185.973006:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435185.973009:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2101f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901696:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435185.973013:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901824:192.168.203.128@tcp:38:kworker.0 00000800:00000100:3.0:1789435185.973027:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435185.973035:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435185/real 1789435185] req@ffff9811d2100e00 x1876350913901824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435185.973046:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2100e00 x1876350913901824/t0(0) o38->lustre-MDT0001-mdc-ffff9811d1e45800@192.168.203.128@tcp:12/10 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435185.973052:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d20d2000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435185.973055:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435185.973058:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2100e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901824:192.168.203.128@tcp:38:kworker.0 00000100:00100000:3.0:1789435185.973063:0:2388:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901952:192.168.203.128@tcp:8:kworker.0 00000800:00000100:3.0:1789435185.973077:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.128@tcp 00000100:00100000:3.0:1789435185.973084:0:2388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789435185/real 1789435185] req@ffff9811d2102a00 x1876350913901952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789435185.973095:0:2388:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9811d2102a00 x1876350913901952/t0(0) o8->lustre-OST0001-osc-ffff9811d1e45800@192.168.203.128@tcp:28/4 lens 520/544 e 0 to 1 dl 1789435210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789435185.973108:0:2388:0:(import.c:65:import_set_state_nolock()) ffff9811d1a44800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789435185.973111:0:2388:0:(import.c:1556:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.128@tcp failed (-110) 00000100:00100000:3.0:1789435185.973114:0:2388:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff9811d2102a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b53d9074-0ce4-44cb-9ce2-c2ae984122e2:2388:1876350913901952:192.168.203.128@tcp:8:kworker.0 Debug log: 78861 lines, 78861 kept, 0 dropped, 0 bad.